Aksal/nuttx
3
1
Fork 1
mirror of https://github.com/apache/nuttx.git synced 2026-08-01 20:28:58 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

doc: Migrating the rest of documentation from cwiki.

Browse source 
* This completes task list in https://github.com/apache/nuttx/issues/11127.
* This preserves selected content from cwiki and moves it to new docs.
* Most pages are simple copy-paste with a simple RST formatting updates,
  with minor updates.
* Content update / reorganization will follow later on when needed.
* Files added (or updated title from cwiki -> current docs):
  * Documentation/implementation:
    * index.
    * cancellation_points.
    * Asynchronous vs. Synchronous Context Switches -> context_switches.rst.
    * ARMv7-M Hardfaults, SVCALL, and Debuggers -> hardfatuls.rst.
    * chip.h FAQ -> chip_h.rst.
    * Debug Output (SYSLOG) Issues -> syslog.rst.
    * Detaching File Descriptors -> file_descriptors.rst.
    * device_nodes.rst.
    * Dynamic Clocking -> power_management.rst.
    * ENOTTY ioctl() Return Value -> ioctl.rst.
    * memory_configurations.rst.
    * kernel_modules_vs_shared_libraries.rst.
    * NAKing USB OUT/IN Tokens -> usb.rst.
    * naming_arch_mcu_board_interfaces.rst.
    * naming_os_internals.rst.
    * nuttx_tasking.rst.
    * oneshot_timers_and_cpu_load.rst.
    * nuttx_initialization_sequence.rst.
    * short_time_delays.rst.
    * Signal Handler Tour -> signal_handlers.rst.
    * smp.rst.
    * syslog.rst.
    * Task Exit Sequence -> nuttx_tasking.rst.
    * tasks_vs_threads.rst.
    * tls.rst.
    * tickless_os.rst.
    * Why Can't Kernel Threads Have pthreads -> kernel_threads_vs_pthreads.rst.
  * Documentation/components/filesystem:
    * smartfs.rst.

Signed-off-by: Tomasz 'CeDeROM' CEDRO <tomek@cedro.info>
This commit is contained in:
Tomasz 'CeDeROM' CEDRO 2026-05-16 13:54:41 +02:00 • committed by Xiang Xiao
parent 2354c720d8
commit 0cfa6f1b96
31 changed files with 8250 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Documentation/components/filesystem/index.rst
View file

@ -1,3 +1,5 @@
.. _nuttx-filesystem:
=================
NuttX File System
=================

4
Documentation/components/filesystem/pseudofs.rst
View file

@ -1,3 +1,5 @@
.. _nuttx-pseudofs:
==================
Pseudo File System
==================
@ -208,4 +210,4 @@ will also fail. You cannot do this, for example:
nsh>
See also NxFileSystem in
`Porting Guide <https://cwiki.apache.org/confluence/display/NUTTX/Porting+Guide>`_
`Porting Guide <https://cwiki.apache.org/confluence/display/NUTTX/Porting+Guide>`_

634
Documentation/components/filesystem/smartfs.rst
View file

@ -478,3 +478,637 @@ Things to Do
- Add reporting of actual FLASH usage for directories (each directory
occupies one or more physical sectors, yet the size is reported as
zero for directories).
Using SmartFS
=============
.. warning:: This section is a copy-paste from old wiki documentation.
It needs review and probably an update.
What is SmartFS
---------------
SmartFS stands for **Sector Mapped Allocation for Really Tiny (SMART) flash**.
It is a filesystem that has been designed to work primary with small,
serial NOR type flash parts that are 1M byte to 16M byte in size
(though this is not a limitation).
The filesystem operates by segmenting the flash (or flash partition)
into "logical sectors" of equal size and then managing them (allocating,
mapping, chaining, releasing, etc.) to build files and directories.
SmartFS Code Layering
---------------------
The system consists of two layers built on top of a standard NuttX MTD driver
layer (with it's associated hardware abstraction layer).
The code directly above the NuttX MTD driver is the SMART MTD layer.
This interfaces with the MTD (flash driver) layer and handles low-level media
operations such as logical sector allocation, freeing and management,
erase block management, low-level formatting, wear leveling, etc.
On top of the SMART MTD layer is the Smart Filesystem code.
The SmartFS code uses the logical sector services of the SMART MTD layer
to provide file and directory level management, such as creating new files,
chaining logical sectors together to create files, creating directories
and file / directory search and management routines.
+------------+------------------------------------+
| Smart FS | fs / smartfs / * |
+============+====================================+
| SMART MTD | ``drivers/mtd/smart.c`` |
+------------+---------+---------+---------+------+
| MTD Driver | m25px | sst25 | filemtd | etc. |
+------------+---------+---------+---------+------+
| HW Driver | spi dev | spi dev | VFS | … |
+------------+---------+---------+---------+------+
Example SmartFS Device Setup
----------------------------
Setting up a device for use with SmartFS is typically done in the config
specific source initialization files and would look something like::
int board_app_initialize(uintptr_t arg)
{
FAR struct spi_dev_s *spi;
FAR struct mtd_dev_s *mtd;
int minor = 0;
/* Initialize the SPI bus #3 with an M25P FLASH driver */
spi = stm32_spibus_initialize(3);
mtd = m25p_initialize(spi);
/* Initialize SMART MTD to work with M25P FLASH device */
smart_initialize(minor, mtd, NULL);
}
Upon successful initialization of the code above,
the NuttX Virtual File System (VFS) will contain a new entry
called ``/dev/smart0`` to represent the SMART MTD device.
Note that this is not a filesystem, but rather a raw block device
(which may or may not already be formatted for use with SmartFS).
To use the ``/dev/smart0`` as a filesystem, it must be initialized
and mounted to the VFS as follows::
nsh> mksmartfs /dev/smart0
nsh> mount -t smartfs /dev/smart0 /mnt
Details of Operation
--------------------
Pages, Blocks, Sectors and things that FLASH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are a number of companies that manufacture FLASH parts, and typically
they use varying terminology in their data sheets when referring
to device geometry.
All devices generally have different region sizes for programming, erasing
and reading and usually use terms like page, erase block, sector
and/or sub-sector.
To avoid confusion, NuttX uses the following terminology:
* **Page or Block:** The smallest area that a device can program
with a single command. This is typically 256 bytes.
* **Erase Block:** The smallest area that a device can erase.
If multiple erase block sizes are supported (e.g. 4K, 32K and 64K), then
the smallest of those sizes, though for larger parts (8M Byte - 32M Byte),
sometimes it is better to use the larger erase block size to keep RAM usage
to a minimum.
* **Sector:** Same as Erase Block.
* **Logical Sector:** A size used by a specific filesystem for managing
a region independently from the actual device specified blocks / sectors.
SmartFS (for example) can use any power of 2 value from 256 through 32768.
Given that each manufacturer and each part has varying geometry sizes,
SmartFS uses a Logical Sector whose size is determined when the device
is formatted (defaulting to ``CONFIG_MTD_SMART_SECTOR_SIZE``).
The SmartFS code then performs all operations using logical sectors and maps
physical accesses to the device based on it's reported geometry.
This allows filesystem performance tuning (total sectors, minimum allocation
size, overhead waste, etc.) independently from the device's erase block size,
etc. Each logical sector contains a 10-byte header (5 for MTD layer,
5 for FS layer) for format management.
An example 128K Byte Flash with 32K and 4K Erase Block sizes and 256 byte page
read/write sizes:
+---------------------+-----------------------------------------------------------------------------------------------------+
| Bulk Erase | Entire Device |
+=====================+========================================+===============+===============+============================+
| Sector Erase | 32K | 32K | 32K | 32K |
+---------------------+-----------------------------+-----+----+----+-----+----+----+-----+----+----+-----+-----------------+
| Sector Erase | 4K | ... | 4K | 4K | ... | 4K | 4K | ... | 4K | 4K | ... | 4K |
+---------------------+-----+-----+-----+-----+-----+-----+----+----+-----+----+----+-----+----+----+-----+-----+-----+-----+
| Page read/write | 256 | 256 | ... | 256 | 256 | ... | 256 | ... | 256 |
+---------------------+-----+-----+-----+-----+-----+-----------------------------------------------------+-----+-----+-----+
| Logical Sector (FS) | 512 | ... | 512 | ... |
+---------------------+-----------+-----+-----------+-----------------------------------------------------------------------+
In the example above, a SMART MTD logical sector size of 512 bytes was chosen.
On the 128K Byte FLASH represented in the table above,
this means there would be:
* 128K / 512 = 256 Logical Sectors Total.
* 4K / 512 = 8 Logical Sectors per Erase Block.
* 512 / 256 = 2 MTD Read/Write Blocks per SMART Logical Sector.
* 10 / 512 = 1.95% Overhead for logical sector headers.
* A maximum of about 250 files / directories supported
(considering format overhead).
* Allocations to files in 512 byte chunks
(a zero-length file consumes 512 bytes).
Choosing a SMART Logical sector size of 256 bytes instead would give
the following results:
* 128K / 256 = 512 Logical Sectors Total.
* 4K / 256 = 16 Logical Sectors per Erase Block.
* 256 / 256 = 1 MTD Read/Write Block per SMART Logical Sector.
* 10 / 256 = 3.9% Overhead for logical sector headers.
* A maximum of about 506 files / directories supported
(considering format overhead).
* Allocations to files in 256 byte chunks
(a zero-length file consumes 256 bytes).
Checking Your MTD Geometry
^^^^^^^^^^^^^^^^^^^^^^^^^^
Most of the NuttX MTD drivers (in the ``drivers/mtd directory``) have been
tested to work with SmartFS / SMART MTD layer and to report block / erase
block sizes correctly.
However there may be one or more drivers that have not been validated.
When configuring SmartFS for the first time using a new / unknown MTD driver,
validate the reported geometries are sane based on the description above.
Additionally, care must be taken when selecting the SMART MTD logical sector
size. The selected size is represented using 3 bits in the logical sector
status byte and stored on each sector.
This means valid values for the logical sector are:
* 256, 512.
* 1024, 2048.
* 4096, 8192.
* 16384, 32768.
As shown in the table / example above, the logical sector size selection
controls the maximum number of files and minimum file size on the volume.
There are some limitations to the value selected imposed by the SMART MTD
code. The logical sector size must be selected such that:
* The total number of sectors on the device / partition fits in a 16-bit word.
This means 65536 or less total sectors (65536 is supported, though
the topmost 2 sectors will never be used).
* The logical sector size cannot be smaller than the MTD device's
block/page size.
* The logical sector size cannot be larger than the MTD devices's
reported erase block size.
* The total RAM used by the SMART MTD layer is dependent on the logical
sector size (when not using Minimize RAM Config option).
The selected logical sector size must not create a RAM requirement greater
than the available RAM.
* The number of logical sectors per MTD device erase block cannot be greater
than 256, and 4 - 128 is the best choice.
* If wear-leveling is enabled, then the following condition must be met:
``Total MTD Erase Blocks / 2 < (Logical Sector Size - 36) * 3``.
Selecting a logical sector size that creates 256 logical sectors per erase
block will create some wasted space on the device.
The SMART MTD layer uses a 1-byte variable for the "free sector count"
and the "released sector count".
This means it can only track up to 255 logical sectors per erase block.
When the sectPerEraseBlk == 256, the last logical sector in each erase
block will never be used, thus causing wasted space on the device.
RAM Usage Calculation
^^^^^^^^^^^^^^^^^^^^^
Efficient management of the filesystem requires building and maintaining RAM
resident status information of the SMART MTD logical sector structure
on the physical device.
During the ``smart_initialize()`` function, the code performs a device scan
(``smart_scan()`` routine) to perform this action.
Tasks performed by the smart_scan are:
* Detect if the format sector exists (logical sector zero with "SMRT" tag).
* Determine the logical sector size that was used to format the device.
* Locate the root directory logical sectors (logical sector 3-11).
* Count the number of free sectors on the device and per erase block.
* Count the number of released sectors on the device and per erase block.
If ``CONFIG_MTD_SMART_MINIMIZE_RAM`` is **not** set:
* Build a map of logical sector to physical sector numbers.
If ``CONFIG_MTD_SMART_WEAR_LEVEL`` is set:
* Allocate wear-level RAM and read leveling info from the format sector.
The amount of RAM consumed is dependent on the config settings
(``MINIMIZE_RAM``, ``WEAR_LEVEL``, etc.).
Rough calculation details are presented for both settings
of ``CONFIG_MTD_SMART_MINIMIZE_RAM``:
When ``CONFIG_MTD_SMART_MINIMIZE_RAM`` is **not** set
+---------------------------+------------------------+-------------------+--------------------+
| Item | RAM Requirement | 1MB / 256 Logical | 8MB / 1024 Logical |
+===========================+========================+===================+====================+
| dev struct | 368-380 (approx) | 376 | 376 |
+---------------------------+------------------------+-------------------+--------------------+
| Logical sector map | total_sectors * 2 | 8192 | 16384 |
+---------------------------+------------------------+-------------------+--------------------+
| Erase block free count | total erase blocks | 16 | 128 |
+---------------------------+------------------------+-------------------+--------------------+
| Erase block release count | total erase blocks | 16 | 128 |
+---------------------------+------------------------+-------------------+--------------------+
| Wear status | total erase blocks / 2 | 8 | 64 |
+---------------------------+------------------------+-------------------+--------------------+
| MTD sector R/W buffer | logical sector size | 256 | 1024 |
+---------------------------+------------------------+-------------------+--------------------+
| FS sector R/W buffer | logical sector size | 256 | 1024 |
+---------------------------+------------------------+-------------------+--------------------+
| Total | **9,176** | **19,128** |
+----------------------------------------------------+-------------------+--------------------+
On larger volumes, the RAM requirement increases significantly because
of the logical sector to physical sector mapping table.
To help keep RAM requirement under control, setting
the ``CONFIG_MTD_SMART_MINIMIZE_RAM`` option eliminates this sector-to-sector
map and replaces it with a sector-to-sector cache.
The cache size is user defined via the
``CONFIG_MTD_SMART_SECTOR_CACHE_SIZE`` option.
Using this RAM reduction mode trades off RAM usage for performance.
The cache always contains the format and root-directory logical to physical
mapping entries, and then stores additional mappings of recently used
logical sectors. When a logical sector is requested that is not contained
in the cache, then the MTD device is scanned front-to-back until
it's physical location on the device is located.
Additionally, if the number of logical sectors per erase block is 16 or less,
then the "free count" and "release count" can be packed into a single byte
per erase block.
When ``CONFIG_MTD_SMART_MINIMIZE_RAM=y`` and
``CONFIG_MTD_SMART_SECTOR_CACHE_SIZE=64``:
+---------------------------+-------------------------+-------------------+--------------------+
| Item | RAM Requirement | 1MB / 256 Logical | 8MB / 1024 Logical |
+===========================+=========================+===================+====================+
| dev struct | 368-400 (approx) | 392 | 392 |
+---------------------------+-------------------------+-------------------+--------------------+
| Logical sector | cache cache entries * 6 | 384 | 384 |
+---------------------------+-------------------------+-------------------+--------------------+
| Free sector bitmap | total sectors / 8 | 512 | 1024 |
+---------------------------+-------------------------+-------------------+--------------------+
| Erase block free count | total erase blocks | 16 | 128 |
+---------------------------+-------------------------+-------------------+--------------------+
| Erase block release count | total erase blocks | 16 | 128 |
+---------------------------+-------------------------+-------------------+--------------------+
| Wear status | total erase blocks / 2 | 8 | 64 |
+---------------------------+-------------------------+-------------------+--------------------+
| MTD sector R/W buffer | logical sector size | 256 | 1024 |
+---------------------------+-------------------------+-------------------+--------------------+
| FS sector R/W buffer | logical sector size | 256 | 1024 |
+---------------------------+-------------------------+-------------------+--------------------+
| Total | **1,832** | **4,168** |
+-----------------------------------------------------+-------------------+--------------------+
Partitions and Mount Points
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creating partitions on the MTD flash / media provides the benefits
of physically isolating one filesystem from another, as well as a mechanism
for creating multiple mount points within the VFS.
There are some pros and cons to consider when deciding to use partitions
with SmartFS:
Partitions - PROS:
* Physically separate filesystems ... possible errors in one
doesn't affect the other.
* Smaller partitions on a large device can use smaller logical sector size
(good if files are small).
* Partitions could be used for redundancy.
* Multiple mount-points within the VFS can be performed if needed.
Partitions - CONS:
* Partition sizes must be understood ahead of time.
* Directory structure / files cannot span across partitions.
* Additional overhead is needed for sector management (one
erase block + 4 pages/blocks reserved per partition
vs. one+4 for entire device).
* Wear leveling (if enabled) will be limited to operate only within
each partition and cannot span multiple partitions globally.
When using partitions with SmartFS, a partition name should be specified
during invocation of the ``smart_initialize()`` routine.
Though not a rule, a suggested standard is to provide partition names
like ``p1``, ``p2``, etc. Then the SMART MTD device entries
in the ``/dev`` directory will take the form::
/dev/smart0p1
/dev/smart0p2
etc.
Prior to the addition of partitions in NuttX, the SmartFS implementation
had already implemented a feature which allows multiple VFS mount-points
to a single SMART MTD device.
This gives the appearance of multiple SmartFS filesystems, though in reality
it simply is a single filesystem with multiple logical root-directories.
Each of the root-directories on the filesystem will be logically isolated
from the others, and after the 'mksmartfs' each will have it's own
``/dev/smart*`` entry.
Multi-Root Directory PROS:
* Supports multiple mount points within the NuttX VFS.
* Each mount-point "appears" to be it's own partition ... each directory
structure is isolated from the others.
* Any mount-point / directory structure can occupy as little or as much
of the filesystem as needed.
* Mount-points can be enabled within partitions.
* Sector management overhead (reserved erase blocks, etc.) are shared
by all mount-points.
* All mount points share RAM structures.
* Wear leveling is performed evenly over all mount-points.
Multi-Root Directory CONS:
* The logical directories are not physically isolated.
Files, directories and individual sectors are all intermixed
on the physical device.
* Any error in the filesystem can effect all mount-points / root directories.
This feature must be enabled specifically using the
``CONFIG_SMARTFS_MULTI_ROOT_DIRS=y`` option.
Setting this option will cause the SMART MTD ``/dev`` entries to be appended
with a directory number, such as ``d1``, ``d2``, etc.
Prior to creating a SmartFS filesystem on the raw device,
a single entry will be identified, such as::
/dev/smart0d1
/dev/smart1p1d1 (device with partitions and multi-root directories)
Then after executing the ``mksmartfs`` command, additional entries
will appear (each of which can be mounted to a different VFS location)::
nsh> ls /dev
...
/dev/ram0
/dev/smart0d1
/dev/zero
nsh> mksmartfs /dev/smart0d1 3
nsh> ls /dev
...
/dev/ram0
/dev/smart0d1
/dev/smart0d2
/dev/smart0d3
/dev/zero
nsh> mount -t smartfs /dev/smart0d1 /data
nsh> mount -t smartfs /dev/smart0d2 /apps
nsh> mount -t smartfs /dev/smart0d3 /recover
The ProcFS Interface
^^^^^^^^^^^^^^^^^^^^
When the PROCFS interface is enabled, each mounted SmartFS device
will appear under::
/proc/fs/smartfs/smart#
The pseudo files reported for each entry will depend on the configured
options. Entries that can currently appear are:
+-------------+-------------------------+---------------------------+
| Entry | ``CONFIG_MTD_SMART_*`` | Meaning |
+=============+=========================+===========================+
| status | | Report volume status |
| | | including geometry. |
+-------------+-------------------------+---------------------------+
| debuglevel | | Write ASCII '0' - '2' |
| | | to set debug print level. |
+-------------+-------------------------+---------------------------+
| erasemap | WEAR_LEVEL=y | Report map (A-N) of erase |
| | | block erasures. |
+-------------+-------------------------+---------------------------+
| mem | ALLOC_DEBUG=y | Print report of all SMART |
| | | MTD memory allocs. |
+-------------+-------------------------+---------------------------+
Example ``procfs`` usage::
nsh> mount -t smartfs /dev/smart0 /mnt
nsh> mount -t procfs /proc
nsh> cat /proc/fs/smartfs/smart0/status
Format version: 1
Name Len: 16
Total Sectors: 4096
Sector Size: 256
Format Sector: 0
Dir Sector: 256
Free Sectors: 4078
Released Sectors: 0
Unused Sectors: 0
Block Erases: 0
Sectors Per Block: 256
Sector Utilization:100%
Uneven Wear Count: 0
nsh> cat /proc/fs/smartfs/smart0/erasemap
BBACAACA
AABAAAAA
nsh>
When Things Don't Work
^^^^^^^^^^^^^^^^^^^^^^
The SmartFS code and MTD layer have been pretty well tested and used
in production products. If things aren't working for you, there are
a couple of places to start debugging first.
Check FLASH MTD Driver
~~~~~~~~~~~~~~~~~~~~~~
Ensure the geometry of the FLASH MTD driver is following the NuttX standard
for block / erase block geometry sizes. Most of them do, but via simple
inspection of the code (as of version 7.16, July 12, 1016), the drivers
that are likely to have improper Geometry reporting (and thus incompatible
with SmartFS) are::
s25fl1.c
sst39vf.c
When configuring to use SmartFS, check the reported geometry of the MTD
driver you are using. If the ``geo.blocksize`` is reported to be the same
as the ``geo.sectorsize``, then there is likely an issue with the MTD driver
implementation.
But it is likely to be a reporting problem of the ``geo.blocksize`` that
is incorrect AND possibly the starting address calculations in the
``_bwrite`` / ``_bread`` routines may be incorrect.
These are BLOCK read / write operations and calculations need to be
performed using the block size (typically 256), not the sector size
(4K, 32K, etc.).
Check CONFIG Options
~~~~~~~~~~~~~~~~~~~~
Double check all of the CONFIG options for both the MTD driver
and the SMART MTD layer:
* ``CONFIG_MTD_SMART_SECTOR_SIZE``: Ensure it's not smaller than the block
size or larger than erase block size.
* ``CONFIG_MTD_BYTE_WRITE``: If enabled, ensure the FLASH actually supports
this mode (single byte programming).
* ``CONFIG_MTD_XXX_SECTOR512``: Ensure this is not set. This should only
be use with FAT volumes.
* ``CONFIG_MTD_XXX_MANUFACTURER``: Double check this with the data
sheet / MTD driver code.
* ``CONFIG_MTD_XXX_MEMORY_TYPE``: Double check this with the data sheet.
Check Writability to the Part
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ensure the FLASH can be written successfully.
If the WP pin is pull active (i.e. the part is write protected),
then the SMART MTD layer will not be able to write any data.
Additionally validate there are no individual protected sectors on the device.
SmartFS Internals
=================
.. warning:: This section is a copy-paste from old wiki documentation.
It needs review and probably an update.
General Structure
-----------------
As described in Using SmartFS, the code is divided into a SMART MTD layer
and a filesystem layer. The MTD layer divides the flash or partition into
equal sized logical sectors, allocates, deallocates and moves them around
as needed to fulfill requests from the filesystem layer.
The filesystem layer uses the logical sectors to store directory and file
information, and to create chains of logical sector numbers to build larger
files (and directories).
The diagram below depicts a portion of a SmartFS device showing the erase
blocks, logical sectors and sector assignments. In the diagram, the top row
of numbers is the absolute sector number within the device.
The bottom row with numbers represents the logical sector number assigned
to each absolute sector.
Using 5 bytes from the header in each sector, the MTD layer manages
the assignments of the logical sector numbers.
The filesystem layer receives the logical sector numbers reported from MTD
layer and assigns them to specific files and/or directories.
As a file grows in size, additional logical sector numbers are requested
and "chained" together. In the diagram, three files are depicted
(files ``a``, ``b`` and ``c``) with varying lengths.
The files shown are as follows:
* File ``a``: Has data in 3 sectors, ``a0-a2``, logical sectors ``12-14``.
* File ``b``: Has data in 4 sectors, ``b0-b3``, logical sectors ``15-18``.
* File ``c``: Has data in a single sector, ``c0``, logical sector ``19``.
+----------------------------------------------------------------------------------------------------+
| Device or partition |
+===================+===================+===================+===================+====================+
| EB | EB | EB | EB | ... |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | ... |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+
| LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | LS | ... |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+
| FS | b0 | -- | -- | RS | b1 | -- | -- | a0 | b2 | -- | -- | a1 | b3 | -- | -- | a2 | c0 | -- | ... |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+
| 0 | 15 | | | 3 | 16 | | | 12 | 17 | | | 13 | 18 | | | 14 | 19 | | |
+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+-----+
* EB = Erase Block.
* LS = Logical Sector.
* FS = Format Sector.
* RS = Root-directory Sector.
* -- = Free Sector.
The filesystem layer uses last 5-bytes of each sector's header to save
the logical sector number of the next logical sector in the chain.
When a file is first created, it's name and beginning logical sector number
are recorded in the Root-directory Sector (or a sub-directory
sector as needed).
The filesystem layer uses the recorded logical sector numbers
in the directory sectors and header sector-chain information to perform
logical sector allocate, read/write and sector release requests to carry out
all typical file system operations.
The SMART MTD layer then performs all the logical to physical mapping,
wear-leveling, sector relocation and block erase operations, etc.
When a sector needs to be physically relocated, it will retain
it's logical sector number, preventing the need to update file sector-chain
information, etc.
The MTD layer will simply update the logical to physical map assignments.
When things change
^^^^^^^^^^^^^^^^^^
Writing data to the filesystem and then reading it back is great.
But at some point you might actually want to change or delete something.
When this happens, the existing data on the flash has to be modified.
As you probably already know, data on a flash can't simply be re-written,
it must be erased in large chunks (erase blocks) that are typically 4K,
32K or 64K in size. When erasing this large a chunk of the flash,
it is highly likely that there will be data in that erase block
which should NOT be erased.
When data in a logical sector on the flash needs to be modified or deleted,
SmartFS simply marks that logical sector as "released" without actually
touching the data. This is done using the characteristic of NOR flash
that allows multiple writes to a given address.
This feature allows any bit of any byte to be changed from a ``1`` state
to a ``0`` state, regardless of whether that byte had previously been written
with other bits set to a ``0`` state.
As long as there is no attempt to change any bits from a ``0`` to a ``1``,
each address can be programmed multiple times.
As shown in the diagrams below, the SMART MTD layer uses the 5th byte
of the logical sector header as a status byte.
The most significant bit of this status byte (``0x80``) indicates
if the sector has been allocated (contains valid data) while the next bit
(``0x40``) indicates if the sector has been "released" (contains data
that is no longer valid). When a sector is first allocated, the
``RELEASE`` bit is held at a ``1`` state, meaning the data
has not been released.
If the data in that sector needs to be changed, deleted or relocated,
the SMART MTD code will simply re-program the status byte,
changing the ``RELEASE`` bit to a ``0`` state.
At that point, all data in that logical sector becomes invalid.
Logical Sector MTD Header
^^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------------------------------------------------------+------------+
| MTD Header (5 bytes) | FS Header |
+=======================+=======+=====+==================================+============+
| Logical Sector Number | Seq # | CRC | Status | 5 Bytes |
+----------+------------+-------+-----+----+----+----+---------+---------+------------+
| LSB | MSB | | | CB | RB | CE | SS(2-0) | FV(1-0) | |
+----------+------------+-------+-----+----+----+----+---------+---------+------------+
* CB: Commit Bit.
* RB: Release Bit.
* CE: CRC Enable Bit.
* SS: Sector Size Bits.
* FV: Format Version Bits.

441
Documentation/implementation/cancellation_points.rst Normal file
View file

@ -0,0 +1,441 @@
.. _cancellation-points:
===================
Cancellation Points
===================
Cancellation Types
==================
In POSIX, there are two pthread cancellation types:
``PTHREAD_CANCEL_DEFERRED`` and ``PTHREAD_CANCEL_ASYNCHRONOUS``.
``PTHREAD_CANCEL_DEFERRED`` is the default and the normal kind of pthread
cancellation that should be used.
You cannot have the ``PTHREAD_CANCEL_DEFERRED`` cancellation type without
cancellation points and you will not have cancellation points unless you
enable them with ``CONFIG_CANCELLATION_POINTS=y``.
``PTHREAD_CANCEL_ASYNCHRONOUS`` is brutal; it simply kills the pthread
immediately without regard for what it is doing.
``PTHREAD_CANCEL_DEFERRED`` works differently; nothing happens immediately.
Instead, the cancellation will be deferred until the pthread is at
(or a better preposition would be within) a cancellation point.
Cancellation points add special instrumentation to a specific set
of interface functions. Those interface functions are listed here:
http://OpenGroup.org.
Enter and Leave Cancelation Point
=================================
In NuttX, each of these functions ``CONFIG_CANCELLATION_POINTS=y`` will enable
a special call to ``enter_cancellation_point()`` at the beginning
of the function and a matching call to ``leave_cancellation_point()`` before
the interface function returns.
These may be nested: One cancellation point function may call another which
may call another, etc. The ``select()`` function, for example, calls
``poll()`` which calls ``sem_wait()`` so the nesting will be three deep.
These two cancellation point hooks behave in a complementary way.
``enter_cancellation_point()`` increments the nesting level and
``leave_cancellation_point()`` decrements the nesting level.
No action is taken unless the nesting level is zero then, in that case,
if there is a pending cancellation then the thread exists normally
by simply calling ``pthread_exit()``.
pthread_cancel()
================
When ``pthread_cancel()`` is called, it will check if the deferred
cancellation mode is enabled for the target pthread. If so it will:
1. Mark the cancellation as pending.
2. If the thread is waiting on a semaphore or message queue (and is in
a cancelable state), it will wake it just in the same was as if
the thread received a single (except with ``ECANCELED``
instead of ``EINTR``).
If, on the other hand, the asynchronous mode is enabled (and the thread is
in a cancelable state), ``pthread_cancel()`` will, instead,
kill the thread immediately.
.. note:: For pthreads, the cancelable state is controlled by the interface
``pthread_setcancelstate().`` This allows the pthread to disable or
re-enable cancelability. If the thread is not cancelable when
``pthread_cancel()`` is called, it will never do more than just mark
the cancellation as pending. When the cancelable state is
re-enabled, then any pending cancellation will take effect.
select()
========
Let's walk through the case of ``select()`` to see how it works:
* When ``select()`` is called, it allocates some memory that it needs to
handle the conversion to ``poll()``.
* It then calls ``poll()`` which sets up the poll with all of the relevant
drivers and, eventually, calls ``sem_wait()``.
So this pthread will spend almost all of its time waiting in ``sem_wait()``
and that is the most likely state of the pthread when ``pthread_cancel()``
is called with deferred cancellation eanbled.
* When ``pthread_cancel()`` is called, it will set the pending cancellation
state and wake up ``sem_wait()`` with the ``ECANCELED`` error.
* ``sem_wait()`` will see the pending cancellation, but will do nothing
because the nesting level is decremented only to two. It will return to
``poll()`` with the ``ECANCELED`` error. Note that the semaphore is left
in a healthy state.
* ``poll()`` will tear-down down the poll from all of the drivers and call
``leave_cancellation_point()`` before it exits. Again, it will see the
pending cancellation but will do nothing because the nesting level
decrements only to 1. Note that since ``poll()`` does the complete
tear-down and all of the drivers are left in a healthy state.
* Finally, ``select()`` cleans up all of its memory allocations and calls
``leave_cancellation_point()``. This time, the nesting level decrements
to zero and ``leave_cancellation_point()`` will call ``pthread_exit()``
with everything in a proper state.
If asynchronous pthread cancellation were selected, then the behavior would
be very different. The first two steps would be the same but then:
* When ``pthread_cancel()`` is called, the thread would be immediately killed.
The semaphore would be left in the locked state; the memory allocated by
``select()`` would be stranded; drivers would be left with stale pointers
to stranded memory. Not a good state to leave the system!
* ``select()`` will not return to the caller in either case.
Application Interfaces
======================
The following pthread application interfaces are available to manage cancellation:
* ``pthread_cancel()`` cancels a thread.
* ``pthread_setcancelstate()`` can enable or disable a cancellation.
* ``pthread_setcanceltype()`` can be used to switched between asynchronous
and deferred thread cancellation.
* ``pthread_testcancel()`` can be used to force a cancellation point.
Task Deletion
=============
NuttX also supports some non-standard interfaces for task deletion
(i.e., cancellation) of tasks as well. The cancellation point logic
is identical and the task application interfaces are very similar:
* ``task_delete()`` deletes (cancels) a task.
* ``task_setcancelstate()`` can enable or disable task cancellation.
* ``task_setcanceltype()`` can be used to switched between asynchronous
and deferred task cancellation.
* ``task_testcancel()`` can be used to force a cancellation point.
Design Issues
=============
.. note:: The issue addressed in this paragraph is covered by
`Issue 619 <https://github.com/apache/nuttx/issues/619>`_
has been corrected in the code base through a series of changes
culiminating in
`PR 733 <https://github.com/apache/nuttx/pull/733>`_.
This discussion still has value, however, as a warning
for things that one needs to take into consideration
when designing logic within the OS.
There is one logical problem in many parts of the system. The following
sequence of code appears in many places in the code base.
It will work not work with thread/task cancellation:
.. code-block:: c
void some_lock(FAR sem_t *sem)
{
while ((ret = nxsem_wait(sem)) < 0)
{
DEBUGASSERT(ret = -EINTR || ret = -ECANCELED);
}
}
That logic will makes it impossible to cancel the thread.
Let me explain why:
* The fact that ``-ECANCELED`` is occurring means that some other task
tried to cancel this one.
* The logic sequence is probably something like:
* OS Function ``A`` is called by the application. Function ``A`` is
a cancellation point. It calls ``enter_cancellation_point()`` on entry
and ``leave_cancellation_point()`` on exit. If the application task is
canceled, then the call to ``leave_cancellation_point()`` is where
the task actually ends (by simply calling exit()).
* OS Function ``A`` calls some internal Function ``B`` which has the above
killer logic in it. It calls ``some_lock()`` and the task is suspended
on waiting to get a count on the semaphore.
When the task is canceled, ``nxsem_wait()`` wakes up with the ``-ECANCELED``
error and returns that error to Function ``B``. But because Function ``B``
just loops and calls ``nxsem_wait()`` again, the task does not terminate;
it just goes back to waiting.
The end result is that Function ``B`` does not return to Function ``A`` so
``leave_cancellation_point()`` is never called and the task is not canceled.
The fix requires a little re-design. The locking function would
have to be like:
.. code-block:: c
int some_lock(FAR sem_t *sem)
{
while ((ret = nxsem_wait(sem)) < 0)
{
DEBUGASSERT(ret = -EINTR || ret = -ECANCELED);
if (ret != -EINTR)
{
break;
}
}
return ret;
}
It then returns a success/failure indication. Calling logic would also have
to be modified to detect the failure case and handle it appropriately.
In the case that the failure case, the calling logic (Function ``B`` in this
example) needs to clean up and return the failure upstream (to Function ``A``)
which must also clean up. Then, eventually, ``leave_critical_section()`` will
be called and the function will exit correctly.
Issue 619 Discussion
====================
.. note:: The
`Issue 619 <https://github.com/apache/nuttx/issues/619>`_
has been corrected in the code base through a series of changes
culiminating in
`PR 733 <https://github.com/apache/nuttx/pull/733>`_.
nxsem_wait_uninterruptible()
----------------------------
In NuttX, thread or tasks may be canceled using ``pthread_cancle()`` or
``task_delete()``. These can also be canceled via certain signals if default
signal actions are enabled.
When threads or tasks are canceled asynchronously, resources may be stranded.
For example, if memory is allocated, it will not be deallocated when the
task/thread is canceled in the FLAT build. These kind on leaks can be handled
by the application with functions registered with ``pthread_cleanup()`` or
``on_exit()``, but others cannot.
The generic solution for NuttX is through the use of cancellation points.
Cancellation points are a mechanism that assures that all resources used
internal by OS interface calls are cleaned-up automatically at designated
cancellation points. An interface is normally a cancellation point if
it has any possibility of blocking.
In NuttX, when threads are blocked waiting of a semaphore when they are
canceled, ``nxsem_wait()`` will return ``-ECANCELED`` and the correct behavior
to let the error ripple back to the cancellation point where the thread/task
will be canceled after all resources have been cleaned up.
Many internal OS functions use ``nxsem_wait_uninterruptible()`` for waiting.
``nxsem_wait_uninterruptible()`` is defined in ``include/nuttx/semaphore.h``.
``nxsem_wait_uninterruptible()`` wraps a call to ``nxsem_wait()`` in
a loop like:
.. code-block:: c
static inline int nxsem_wait_uninterruptible(FAR sem_t *sem)
{
int ret;
do
{
/* Take the semaphore (perhaps waiting) */
ret = nxsem_wait(sem);
}
while (ret == -EINTR || ret == -ECANCELED);
return ret;
}
The correct behavior is to continue waiting on ``-EINTR`` only.
If the task/thread is awakened by a signal just continue waiting.
The behavior for ``-ECANCELED`` is not correct. It should not loop.
Looping like this will prevent the cancellation from working.
Instead of returning through the cancellation point, the code will be stuck
in the above loop (in defferred cancellation mode) and cannot be cancelled.
The correct behavior is to return when ``-ECANCELED`` is encountered,
not to continue looping:
* GOOD: ``while (ret == -EINTR);``
* BAD: ``while (ret == -EINTR || ret == -ECANCELED);``
The only complexity to this suggested change is that the callers of
``nxsem_wait_uninterruptible()`` must also check the return value for any
errors and make sure that that error is returned to the caller.
The error must propogate all the way back up to the cancellation point.
These are other related inline functions in ``include/nuttx/semaphore.h``
that behave in this same way. This issue applies to them as well.
.. important:: If cancellation points are enabled, the default cancellation
mode should be deferred mode.
.. note:: There are three interfaces affected by this issue:
``nxsem_wait_uninterruptible()`` as discussed above, but also
``nxsem_timedwait_uninterruptible()`` and
``nxsem_tickwait_uninteruptible()`` which are closely related.
Mutual Exclusion Semaphores vs. Signaling Semaphores
----------------------------------------------------
Semaphore have two usages in NuttX:
* Mutual Exclusion Semaphores: Protect the shared data and force mutually
exclusive use of semaphores.
* Signling Semaphores: Used to wait for events to be signaled asynchronously.
Should we return ``-ECANCEL/-EINTR`` for both? Or only for ithe second?
I think both should work the same: For both the semaphore wait should be
terminated and correct error should be returned.
Mutual Exclusion Semaphores
---------------------------
If the API requires returning ``EINTR`` if a signal is received, we should
return the error. This is required in many POSIX interfaces and don't see
how you could avoid that. However, the mutual exclusion semaphores are
the just there for the correctness in the design. There is seldom any real
competitor for the exclusion semaphore so, in general, they do not block and,
hence, would never generate any error.
And even if taking the mutual exclusion semaphore does cause the thread
to block, it should block for only a very brief moment of time and the
likelihood of a signal received or of the task being canceled is very small.
But if there is even a small possibility of that event happening,
it WILL happen in an embedded system (Murphy's law).
But if during the that brief moment while the thread is blocked,
a signal is received, I think that it is correct to return ``-EINTR``.
This is not so critical only because such an event it is rare.
But the specification requires it so we should do it.
When to Return ECANCELED
^^^^^^^^^^^^^^^^^^^^^^^^
The ``ECANCELED`` error is **ABSOLUTELY** critical to return under any
circumstance. That cancellation notification is received only once and
it if is ignored, then the thread will not cancel it will continue to run.
It really must return immediately with ``ECANCELED`` for the cancellation
to work quickly and reliably, where ever ``ECANCELED`` is detected.
Ignoring it is a hard bug in any case.
Signaling Semaphores
^^^^^^^^^^^^^^^^^^^^
The signaling semaphores are a little different story. They may block
for a very long time, for example, waiting for the receipt of data that
may never come. For example, a read from a serial port will hang indefinitely
if nothing is received on the serial port. So if a signal is received or
if the thread is canceled, then it is essential to terminate the semaphore
wait and return the error condition.
We should have no difference of opinion on that.
There is really no difference in the actions that must be taken if a signal
is received of if the task is canceled. Both mutual exclusion semaphores
and signaling semaphores should wake up and return the error condition.
The only real differences is that mutual exclusion semaphores either
do not block or, if they do, the do not block as long.
Mutual Exclusion Semaphores and Serialization
---------------------------------------------
There are situations were the task may block for a very long time,
even indefinitely, on a mutual exclusion semaphore. Consider the following
scenario: A driver that reads data could have a sequence like this:
1. Get exclusive access to the driver by taking a semaphore.
2. Wait on a signaling semaphore for data (might be a long time).
3. Release the mutual exclusion semaphore.
Now suppose there are two applications, Task ``A`` and Task ``B``,
that open the driver and try to read from it. The first, Task ``A``, will get
the mutual exclusion semaphore at ``(1)`` and then will hang waiting
for data at (2). It may have to wait for a very long time to receive data.
When re-entered by the second, Task ``B``, it will hang at ``(1)``
also for very long time. Task ``B`` is effectively queued and cannot event
begin its wait on the signaling semphore until Task ``A`` receives its data
and releases the mutual exclusion semaphore. This is normal behavior
and exactly, how this kind of driver is supposed to work:
This use of mutual exclusion impelements serialization of I/O.
But now suppose that Task ``B`` receives a signal interrupt or a cancellation
event while waiting on the mutual exclusion semaphore. It MUST terminate the
wait and return ``-EINTR`` or ``-ECANCEL`` immediately. Task ``A`` may never
receive data and if that action is not taken, the signal interrup
or cancellation event is lost and the functionality has failed.
When to Return EINTR
^^^^^^^^^^^^^^^^^^^^
Returning ``EINTR`` really only applies to ``read()`` and ``write()``
(and ``open()``) functions as covered by Issue #669 . In other places,
``EINTR`` can be ignored. ``open()`` and ``close()`` should also
return ``EINTR``. But I don't think that is very meaningful for ``close()``
since most people ignore the return value from ``close()``.
But, on the other hand, ``ECANCELED`` can never be ignored under any
condition.
Detecting, Remembering, and Propagating Signal Interrupts and Cancellation Events
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Currently, the only way for an OS interface function to know if a signal
was received is to be awakened with an ``EINTR`` error. That is why
it is critical to always return the ``EINTR`` error and conform with
the POSIX requirements.
However, if a signal is received while the OS interface is NOT waiting,
then there is no way to know if the signal was received. I think that
is a limitation in the current design. There probably should be some latching
indication, perhaps in the TCB, that a signal interrupt has been received.
There is already a ``TCB_FLAG_CANCEL_PENDING`` in the TCB that will tell us
that the if the task has been canceled. That flag is tested in the
``leave_cancellation_point()`` function so I don't think that there is any
corresponding issue for thread cancellation. We just need to make sure that
all waits are aborted if ``ECANCELED`` is received and let the error
indication ripple all they back to the ``leave_cancellation_point()``
function. Then the function will exit cleanly, safely, and quickly.
Interestingly, the ``ECANCELED`` error will never be seen by the application.
It just triggers the return uwind sequence where all resources are recovered
and finally until ``leave_cancellation_point()`` is called. Then the thread
will exit before it returns to the applicaton.
You can see all of this working in the ``board/sim/sim/sim/configs/ostest``
configuration if you also enable::
CONFIG_CANCELLATION_POINTS=y
CONFIG_PTHREAD_CLEANUP=y
There is a ``cancel.c`` test within the OS test, but the more interesting test
is the ``pthread_cleanup.c`` test. You can see how all this works when the code
unwinds with the ``ECANCELED`` error and calls ``leave_cancellation_point()``.
Just single step through ``pthread_cond_wait()`` to see this.
``pthread_cond_wait()`` is a ``cancellation_point()`` and that is where
the thread exit will occur.
`PR#749 <https://github.com/apache/nuttx/pull/749>`_ adds those settings
to that ``sim:otest`` defconfig.

125
Documentation/implementation/chip_h.rst Normal file
View file

@ -0,0 +1,125 @@
.. _chip-h:
======
chip.h
======
The purpose of the two chip.h files in each arm chip
====================================================
If you wonder about the purpose of the two ``chip.h`` files in each arm chip.
.. code:: sh
$ find arch/arm -name chip.h | grep stm32
arch/arm/include/stm32/chip.h
arch/arm/src/stm32/chip.h
The reason behind ``arch/arm/src/stm32/chip.h`` file was a bad idea
that happened a long time ago.
Right now, I believe that its only required when ``CONFIG_ARMV7M_CMNVECTOR``
is selected in the configuration. In that case, ``arch/arm/src/stm32/chip.h``
is included by ``arch/arm/src/armv7-m/up_vectors.c`` in order provide
the number of interrupt vectors. In stm32, ``arch/arm/src/stm32/chip.h``
provides the number of vectors indirectly by including the correct,
chip-specific vectors.h file.
This function is a little more obvious in ``arch/arm/srch/lpc43xx/chip.h``.
This ``arch/arm/src/xyz/chip.h`` is also a good way to export awkward
internal header files in a cleaner way. But that use is optional
and not required outside of the chip- and board-related directories.
For the ``arch/include/stm32/chip.h`` file, only set of definitions
is required there, the NVIC priorities:
* ``NVIC_SYSH_PRIORITY_MIN`` provides the lowest interrupt priority
supported by the system. This is the highest numeric value;
but the lowest interrupt priority.
* ``NVIC_SYSH_PRIORITY_DEFAULT`` provides the default priority of all
interrupts. Since truly nested interrupts are not supported, this is
the priority of all interrupts except for the SVCall interrupt.
* ``NVIC_SYSH_PRIORITY_MAX`` provides the maximum interrupt priority.
For all ARMv7-M's, this will be the value zero.
* ``NVIC_SYSH_PRIORITY_STEP`` is the value need to increment from one
priority level to next, lower priority level.
If ``CONFIG_ARMV7M_USEBASEPRI`` is selected, then interrupts will be disabled
by setting the ``BASEPRI`` register to ``NVIC_SYSH_DISABLE_PRIORITY`` so that
most interrupts will not have execution priority.
SVCall must have execution priority in all cases.
In the normal cases, interrupts are not nest-able and all interrupts run
at an execution priority between ``NVIC_SYSH_PRIORITY_MIN`` and
``NVIC_SYSH_PRIORITY_MAX`` (with ``NVIC_SYSH_PRIORITY_MAX`` reserved
for SVCall).
If, in addition, ``CONFIG_ARCH_HIPRI_INTERRUPT`` is defined, then special high
priority interrupts are supported. These are not "nested" in the normal sense
of the word. These high priority interrupts can interrupt normal interrupt
processing but execute outside of OS (although they can "get back
into the game" via a PendSV interrupt).
In the normal course of things, interrupts must occasionally be disabled
using the ``up_irq_save()`` inline function to prevent contention in use
of resources that may be shared between interrupt level and non-interrupt
level logic. Now the question arises, if we are using the ``BASEPRI``
to disable interrupts and have high priority interrupts enabled
(``CONFIG_ARCH_HIPRI_INTERRUPT=y``), do we disable all interrupts except
SVCall (we cannot disable SVCall interrupts)?
Or do we only disable the "normal" interrupts?
If we are using the ``BASEPRI`` register to disable interrupts, then
the answer is that we must disable ONLY the normal interrupts. That is
because we cannot disable SVCALL interrupts and we cannot permit SVCAll
interrupts running at a higher priority than the high priority interrupts.
Otherwise, they will introduce jitter in the high priority interrupt
response time.
Hence, if you need to disable the high priority interrupt, you will have to
disable the interrupt either at the peripheral that generates the interrupt
or at the interrupt controller (e.g., ``NVIC``). Disabling global interrupts
via the ``BASEPRI`` register cannot effect high priority interrupts.
* ``NVIC_SYSH_MAXNORMAL_PRIORITY`` is the maximum priority of "normal"
interrupts. Interrupt are disabled at the level
``NVIC_SYSH_MAXNORMAL_PRIORITY``, disabling all "normal" interrupt
but leaving the high priority and SVCALL interrupts enabled.
* ``NVIC_SYSH_HIGH_PRIORITY`` is the priority of the high priority interrupt.
* ``NVIC_SYSH_DISABLE_PRIORITY`` is the level at which interrupts will
be disabled. This will be equal to ``NVIC_SYSH_MAXNORMAL_PRIORITY``.
* ``NVIC_SYSH_SVCALL_PRIORITY`` is the priority of the SVCall interrupt.
This must be higher priority than ``NVIC_SYSH_DISABLE_PRIORITY`` so that
it is never disabled, but lower in priority than
``NVIC_SYSH_DISABLE_PRIORITY`` so that SVCall handling cannot interfere
with high priority interrupt handling.
These definitions only have to be here because NVIC implementations may differ
in the number of priority levels.
Now, by convention, the ``arch/xyz/include/chip/abc/chip.h`` file is also used
to provide all of the definitions that discriminate the features of the
different members of the chip family.
It is not technically necessary that those chip feature definitions exist
in this header file, but it is a good convention to follow so that
it is easier for people who have to work across the differ architectures.
There is another really big difference between the two chip header files:
They also different in scope:
1. Code in the ``arch/arm/src/xyz`` and in ``boards/arm/xyz/abc/src``
directories can both include the ``arch/arm/src/xyz/chip.h`` header file,
but nothing else can (only because nothing else is provided the header
file path in the build system). So that ``chip.h`` file is intended to be
used only in low-level board/chip logic.
2. The ``chip.h`` header file at ``arch/arm/include/xyz/chip.h``,
on the other hand, will be linked at include/arch/chip and, hence,
can be included anywhere in the system, even from applications.
It can be included like::
#include <arch/chip/chip.h>
So the ``arch/arm/include/xyz/chip.h`` header file is a place where you would
want to put chip-related stuff that can be made visible to application code.
Chip capabilities is a good example of the kind of knowledge that
applications might care about.

87
Documentation/implementation/context_switches.rst Normal file
View file

@ -0,0 +1,87 @@
.. _context-switches:
================
Context Switches
================
Two Types of Context Switches
=============================
There are really two different kinds of context switches.
We refer to them as synchronous and asynchronous context switches (but there
might be better names):
* **Synchronous context switch** occurs when the system is interrupted and,
because of on actions within the interrupt handler, a context switch is
generated. In this case, the state of the interrupted task is saved when
the interrupt handler is entered but a different task state is restored
when the interrupt handler returns.
* **Synchronous context switch** in our terminology occurs when a task
explicitly suspends itself by calling some OS interface that causes
the task to block, such as ``usleep()``.
Synchronous Context Switches
============================
There are two ways to implement a synchronous context switch:
``up_savecontext()`` and ``up_fullcontextrestore()``.
You can implement the moral equivalent of ``setjmp()`` and ``longjmp()``
on steroids. Some architectures have a function called ``up_savecontext()``
that is the moral equivalent of ``setjmp()``, it saves the current state
of the task (and like ``setjmp()`` returns ``0`` or ``1`` to indicate
if the context is being restored.
Another function ``up_fullcontextrestore()`` is like ``longjmp()``,
it restores the context saved by either the interrupt handler during
a previous asynchronous context switch or by the ``up_savecontext()``.
The naming differences ``save`` vs ``fullrestore`` is because when
``up_savecontext()`` is called, it does not need to save all of registers.
Only a subset needs to be saved because the processor ABI provides that some
registers are volatile or caller-saved when ``up_savecontext()`` is called.
The are a couple of downsides to the this approach.
First, the ``up_savecontext()`` and ``up_fullcontextrestore()`` functions
are tricky to write. Second, they have limited usage.
They can be used only in the FLAT build mode where all tasks are running with
the same privileges. If you were to try to do ``up_fullcontextrestore()``
to get from an unprivileged task to a privileged task, you would get an
access violation exception of some sort.
System Calls
============
In order to the limitations of ``up_savecontext()`` and
``up_fullcontextrestore()``, you have to do something a little differently.
One way is to use a system call (a **software interrupt**, a **trap** in x86
or an **SVCALL** in ARM land). This generates a software interrupt and uses
the mechanization of the asynchronous context switch: The software interrupt
saves the context of the old task on entry (replacing the functionality of
``up_savecontext()``) and restores the new task context on return (replacing
the functionality of ``up_fullcontextrestore()``).
The tiny software interrupt handler just sets up the context switch.
The ARMv7-M does synchronous contest switches in this way. You can see how
this is done in the ARMv-7M the SVCALL software interrupt handler.
The advantages of this approach are:
1. It is trivially easy to implement. If interrupt level asynchronous context
switches work, then so will these synchronous context switches.
2. You an switch between privileged and unprivileged tasks.
That happens for free when the interrupt returns.
The downside is only that it causes significantly slower synchronous context
switching times. It adds the overhead of interrupt processing and interrupt
IRQ dispatching to each context switch.
I think this is still the correct way to go despite its worse performance.
Several modifications would be required to convert from the first to the
second type of synchronous context switches:
1. Creation of the software handlers for the context switch.
2. Converting all of the back-to-back calls to ``up_savecontext()`` and
``up_restorefullcontext()`` to a single call to a function that executes
the software interrupt, usually something like ``up_switchcontext()``.

94
Documentation/implementation/device_nodes.rst Normal file
View file

@ -0,0 +1,94 @@
.. _device-nodes:
============
Device Nodes
============
Linux Device Nodes
==================
I used to have good Linux expertise a decade or so ago.
But my current Linux knowledge is dated and rusty.
I don't know anything about udev, SystemD, devtmpfs, sysfs, or any of that.
So this is my simplified understanding.
Device files work quite a bit differently in Linux and NuttX.
A device node in Unix/Linux only really contains the only type of the device
and its major and minor device numbers, i.e., it just holds data.
So creating the device node does not install or create the driver;
it simply writes a tiny file containing some special data.
Nothing happens until you try to open the device.
If something in the operating system has not initialized and registered
a driver for that type and major/minor numbers,
then you fail to open the device.
So the device nodes and the device drivers are decoupled in Linux/Unix
and there is a rendezvous that must occur later for the device node
to actually refer to the device.
"(..) Linux maps the device special file passed in system calls
(say to mount a file system on a block device) to the device's
device driver using the major device number and a number of system
tables, ...The major number is actually the offset into the
kernel's device driver table, which tells the kernel what
kind of device it is (whether it is a hard disk or a serial
terminal) (..)"
-- Source:
www.linux-tutorial.info/modules.php?name=MContent&pageid=94.
Normally, when you create a Linux file system, you also create all of the
standard device nodes. But most of these do not map to real devices.
If you try to access most of the devices under ``/dev`` in Linux, they will
fail because the underlying driver that maps to that major/minor number
has not been initialized.
NuttX Device Nodes
==================
NuttX does not use major/minor device numbers and there are no device
"system tables" to associate major/minor numbers to a driver implementation.
NuttX simplifies this be removing the "man in the middle": When you register
the driver, you also create the device node.
.. important:: The device node IS the driver registry.
This is a tremendous simplification and one of the things that
makes NuttX usable in the constrained MCU environment.
In NuttX, device nodes are not really files at all.
They are special entries in the NuttX root pseudo-filesystem.
See :ref:`NuttX Pseudo File System <nuttx-pseudofs>` for more details.
Usage Differences
=================
.. important:: Only devices drivers can create device nodes and the existence
of the device node means that the device has been initialized,
registered, and is ready for use (with the exception of some
removable devices that may not actually be ready).
You cannot create device nodes from applications!
You could argue that this simplification is a deviation from my Unix/Linux
roadmap and would have to agree that you are right.
But it is also the kind of enabling simplification that makes a tiny
Unix-like operating system feasible on these lower end MCUs.
In Linux standard device drivers are initialized and registered as with NuttX.
A (privileged) application can create a device node, but cannot initialize
or register a device driver directly (as far as I know).
I believe that if you want to instantiate an uninitialized, unregistered
device driver you would have to install a kernel module containing
the driver (which would probably also create the device nodes corresponding
to the driver).
boardctl()
==========
NuttX does support a sneak interface to support interactions with board-level
OS logic. That sneak interface is ``boardctl()`` (see :ref:`board-ioctl` and
:ref:`nuttx-initialization-sequence` for more details).
That interface could potentially be used to force initialization of device
drivers by application code. That discussion is to be provided.

231
Documentation/implementation/file_descriptors.rst Normal file
View file

@ -0,0 +1,231 @@
.. _file-descriptors:
================
File Descriptors
================
Drivers within Drivers
======================
Have you ever wanted to write a driver the uses another driver?
For example, suppose you want to develop a character driver for a module
that provides a serial interface to the host. Wouldn't you like to be able to
open and use the serial driver within your module driver?
Or suppose the you have a device with analog inputs such as joystick
and you would like to use the ADC character driver to sample the joystick?
Would you not want to contain the ADC character driver within the joystick
driver?
Drivers File Descriptors
========================
One obstacle to containing a driver instance within another driver
is the nature of the file descriptor. When you open a file or a driver from
any thread in a task, you receive a file descriptor that you then may use
in the task to access the device.
In the NuttX implementation, that file descriptor is really an index into
a pre-allocated array of type struct file.
It is really the underlying struct file instance referred to by the
file descriptor that use used to access the device.
The file descriptor/index is simply a portable way to reference the struct
file instance.
.. note:: In NuttX, threads are organized into task groups, see
:ref:`Tasks vs. Threads <tasks-vs-threads>` for more details.
Each task group has its own array of struct file. All of the threads within
the same task group share the same array allocation. Therefore, the same file
descriptor index is valid for all threads within the task group.
But, on the other hand, threads running within a different task group
use a different array of struct file and file descriptors opened
in this other task group have no relationship.
In short, file descriptors cannot be used by different tasks.
A driver must not be associated with any specific task group's resources.
That would make the driver dependent upon that task group. Any task should be
able top open and use the driver. Therefore, the driver cannot use
file descriptors to access the contained driver instance.
So, if you cannot use file descriptors within a device drivers, how can
one driver contain another driver instance? The answer is by detaching
the file descriptor from the open file instance.
Detaching Open Files
====================
There is an internal OS interface called ``file_detach()`` that is prototyped
and described in the header file ``include/nuttx/fs/fs.h``:
.. code-block:: c
/****************************************************************************
* Name: file_detach
*
* Description:
* This function is used to device drivers to create a task-independent
* handle to an entity in the file system. file_detach() duplicates the
* 'struct file' that underlies the file descriptor, then closes the file
* descriptor.
*
* This function will fail if fd is not a valid file descriptor. In
* particular, it will fail if fd is a socket descriptor.
*
* Input Parameters:
* fd - The file descriptor to be detached. This descriptor will be
* closed and invalid if the file was successfully detached.
* filep - A pointer to a user provided memory location in which to
* received the duplicated, detached file structure.
*
* Returned Value:
* Zero (OK) is returned on success; A negated errno value is returned on
* any failure to indicate the nature of the failure.
*
****************************************************************************/
#if CONFIG_NFILE_DESCRIPTORS > 0
int file_detach(int fd, FAR struct file *filep);
#endif
In order to use ``file_detach()`` you would do the following:
* Allocate an instance of struct file either dynamically or statically.
This instance will represent the contained driver.
* Open the file or driver to get a file descriptor
* Call ``file_detach()`` in order to clone the underlying struct file to your
allocated instance. Detaching the file descriptor has the side-effect
of closing the original instance.
* You can then use the driver methods of the contained driver directly from
your driver. These driver methods are described in ``include/nuttx/fs/fs.h``.
There is also a ``file_close_detached()`` internal OS function that you can use
to recover resources if you no longer need the contained driver instance:
.. code-block:: c
/****************************************************************************
* Name: file_close_detached
*
* Description:
* Close a file that was previously detached with file_detach().
*
* Input Parameters:
* filep - A pointer to a user provided memory location containing the
* open file data returned by file_detach().
*
* Returned Value:
* Zero (OK) is returned on success; A negated errno value is returned on
* any failure to indicate the nature of the failure.
*
****************************************************************************/
int file_close_detached(FAR struct file *filep);
This technique is not limited to character drivers but can also be used
with files in a file system or even block drivers.
Detached File Helpers
=====================
Once the file structure has been detached from its file descriptor,
you can no longer use the standard VFS functions ``read()``, ``write()``,
``ioctl()``, etc. Fortunately, there are a parallel set of interfaces
that can be used with detached files. These are decribed in detail
in ``include/nuttx/fs/fs.h`` and only listed here below:
.. code-block:: c
ssize_t file_read(FAR struct file *filep, FAR void *buf, size_t nbytes);
ssize_t file_write(FAR struct file *filep, FAR const void *buf, size_t nbytes);
ssize_t file_pread(FAR struct file *filep, FAR void *buf, size_t nbytes,
off_t offset);
ssize_t file_pwrite(FAR struct file *filep, FAR const void *buf,
size_t nbytes, off_t offset);
off_t file_seek(FAR struct file *filep, off_t offset, int whence);
int file_ioctl(FAR struct file *filep, int req, unsigned long arg);
int file_fsync(FAR struct file *filep);
int file_dup2(FAR struct file *filep1, FAR struct file *filep2);
int file_vfcntl(FAR struct file *filep, int cmd, va_list ap);
The SYLOG Device: A Case Study
==============================
This technique is used for the SYSLOG device. Originally, NuttX used
file descriptor ``1`` for SYSLOG output by default. For most task groups,
file descriptor ``1`` (``stdout``) mapped to ``/dev/console`` and this
solution worked well in most cases.
But using file descriptor ``1`` caused some very strange results when
``stdout`` was redirected such as when a USB or a Telnet console was used.
In those cases, SYSLOG output would go to different places depending upon
how stdout was re-directed within a given task group.
This was fixed using ``file_detach()``. In the default case, the SYSLOG
initialization logic now opens ``/dev/console`` then calls ``file_detach()``
to disassociate the open file instance from any task group.
Now, SYSLOG output consistently goes to ``/dev/console`` regardless of how
file descriptor ``1`` may be re-directed when SYSLOG output is generated.
Other Examples
==============
There are some other examples in analog joystick lower half drivers
that use the ADC character driver to read joystick positions::
boards/arm/stm32/nucleo-f4x1re/src/stm32_ajoystick.c:
ret = file_detach(fd, &g_adcfile);
boards/arm/stm32l4/nucleo-l476rg/src/stm32_ajoystick.c:
ret = file_detach(fd, &g_adcfile);
boards/arm/sama5/sama5d3-xplained/src/sam_ajoystick.c:
ret = file_detach(fd, &g_adcfile);
Socket Descriptors
==================
There is a similar story for socket descriptors but this is probably
not the place for that whole story. Instead, we will just summarize
that story here. First, let's compare file and socket descriptors here.
Socket descriptors are similar to file descriptors in that:
* They are a numeric index into a task-private array of structures.
For the case of socket descriptors, this is an array of type struct socket.
* Like the file descriptor, the socket descriptor is simply a portable way
to reference the underlying socket structure.
* And also like the file descriptor, the socket descriptor is meaningful
only in the context of task group where the socket descriptor was created.
As in the case for file descriptors, there are also a set of socket interfaces
that accept a reference to struct socket as an input parameter.
For each standard socket interface that accepts a socket descriptor,
there is a corresponding non-standard, OS-internal interface that accepts
a reference to struct socket as as an input parameters. These low-level socket
interfaces all have the prefix ``psock_`` and are all prototyped in the file
``include/nuttx/net/net.h``. They are not discussed further here.
These low-level socket interfaces are also independent of task groups
and may be used without regard to the thread that created the socket.
The primary usage difference is in how the struct socket instances are created:
* In the case of struct file, there is no special ``file_open()`` interface
to create the detached file reference. Instead, a two step procedure is used:
First the file is opened the standard ``open()`` function to obtain the
file descriptor, then the struct file is detached from the file descriptor
using ``file_detach()``.
* The struct socket instance, on the other hand, is created in the detached
state directly using ``psock_socket()``.
You can find a good example of a character driver that contains
a task-independent socket interface in the Telnet character driver at
``driver/net/telnet.c``. That driver is used to encapsulate a Telnet
session and to provide Telnet ``stdin``, ``stdout``, and ``stderr``
for remote NSH sessions.

144
Documentation/implementation/hardfaults.rst Normal file
View file

@ -0,0 +1,144 @@
.. _hardfaults:
==========
Hardfaults
==========
ARMv7
=====
Cortex-M3 and Cortex-M4
-----------------------
The most popular CPUs in current MCU designs are the Cortex-M3 (ARMv7-M) and
the Cortex-M4 (ARMv7E-M). Handling of these two architectures is almost
identical in NuttX (unless hardware floating point is enabled).
SVCALL
------
NuttX uses the SVCALL software interrupt in order to perform certain steps
in the context switching for the Cortex-M3 and Cortext-M4.
This sequence of logic appears in several places:
* Create a short critical section by disabling exceptions,
* Perform some set-up,
* Initiate the software exception / SVCALL, and
* When the software exception processing returns, re-enable exceptions.
.. note:: There is a technical difference between interrupts and exceptions.
In this section the term exception will be used. It is probably
more accurate since interrupts are really exceptions that result
from device interrupt lines. Furthermore, when we refer to
exceptions in this section, we are referring specifically to
ARMv7-M configurable exceptions.
Disabling Interrupts via the PRIMASK register
---------------------------------------------
The ARMv7-M architecture supports a register called the ``PRIMASK`` register.
The ``PRIMASK`` register contains a single valid bit. If that bit is set to
one, then exceptions are disabled. If that bit is zero, exceptions are enabled.
More correctly, when this bit is set to one, it prevents the activation of all
exceptions with configurable priority.
The original NuttX implementation used this ``PRIMASK`` register to enable
and disable exceptions. Things now get interesting in the sequence of logic
listed above because the ``PRIMASK`` bit also disables the SVCALL exception!
So, instead of taking the ``SVCALL`` exception vector, the Cortex-M3/4
generates a hardfault exception (see ARM.com's discussion of
`Activation Levels <https://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0337g/Chdbdfjf.html>`_).
These hardfaults are not really a problem. The design of the NuttX hardfault
handler expects these exceptions and does the right thing.
However, the occurrence of hardfaults may come as a surprise to many people
and especially to some debuggers.
Hardfaults and Debuggers
------------------------
These hardfaults only become a technical issue when dealing with a debugger.
What does the debugger do when the hardfault occurs?
We need to back off and think about some system philosophy here.
The deep philosophical question here is: Who is in charge of system integrity?
The debugger or the RTOS?
If you are running a primitive NoOS program (like the famous blinky test
program), then you are running a barebones system and you need
all of the help you can get. So having the debugger make decisions about what
is the proper behavior of the blinky program and what is not is a good
thing for you.
But if you are using an advanced RTOS, then the RTOS will want to take
responsibility of the health of your system and now there is the possibility
of inconsistencies between the decisions that the RTOS makes and the decisions
that your debugger makes this hardfault handling is a perfect example here.
In NuttX, the hardfaults are controlled by the RTOS, but some debuggers will
break when the hardfault occurs and make debugging impossible.
Some people might take issue with this. Breaking on hardfault can make
debugging easier, since the break happens in the throwing context and you have
a hope of obtaining a backtrace and debugging the throwing side callstack.
However, I would suggest that putting a break point on ``up_assert()`` would
accomplish the same thing without being so intrusive.
How did the debugger know that the hardfault occurred? It knew because of
settings in the "ARM's Debug Exception and Monitor Control Register" or
``DEMCR``. Proper settings of the ``DEMCR`` register will allow debugger
to get break exceptions when a hardfault occrs.
So one workaround is to just reconfigure the ``DEMCR`` register so that break
exceptions are no longer generated when hardfaults occur.
Here is an example of such logic for the LPC43xx MCU.
Decoupling the hardfault from the break exception in this way does not work
with all debuggers, however. Presumably because some debuggers re-enable break
exceptions on hardfaults.
Disabling Interrupts via the BASEPRI register
---------------------------------------------
The ARMv7-M architecture supports another way to disable exceptions using
a register called the ``BASEPRI`` register. ARMv7-M exceptions are prioritized.
Each exception can be assigned an 8-bit priority. If the ``BASEPRI`` register
is set to a non-zero priority value, then it will filter exceptions in this sense:
* Exceptions with priority lower than or equal to the ``BASEPRI`` register
will be disabled.
* Exceptions with priority higher than the ``BASEPRI`` register will still
be enabled.
Normally this interrupt prioritization is used to support nested interrupt
handling, but it can also be used for disabling of all exceptions
if configured properly.
.. note:: In the ARMv7-M, higher values correspond to lower priority.
This can be really confusing!
NuttX supports a configuration option called ``CONFIG_ARMV7M_USEBASEPRI``.
If this option is selected, then the exception prioritization and control
logic will be configured to use the ``BASEPRI`` register instead of the
``PRIMASK`` register to disable exceptions.
This configuration includes the following changes in the behavior:
* Normal interrupts and exceptions are restricted to the range
``{ lowest priority ... (highest priority - 1) }``.
* The priority of the ``SVCALL`` exception is set to highest priority.
* When exceptions are enabled, the ``BASEPRI`` register is set to zero,
enabling exceptions of all priorities.
* When exceptions are disabled, the ``BASEPRI`` register is set to
``(highest priority - 1)``, disabling all exceptions except for the
``SVCALL`` exception.
In this way, the ``SVCALL`` exception remains enabled when exceptions
are disabled and no hardfault occurs.
.. note:: The above is inaccurate on several counts. It was simplified
to make the discussion sane. Not only do higher values correspond
to lower priorities, but the increment between consecutive 8-bit
priority values is probably not one. The supported maximum and
minimum priority values (as well as the step in each priority value)
may be different for each MCU. To handle this, these values are
exported in NuttX for each ARMv7-M architecture by header files at
``nuttx/arch/arm/include/<chip>/chip.h``.

38
Documentation/implementation/index.rst
View file

@ -6,12 +6,36 @@ Implementation Details
:maxdepth: 2
:caption: Contents:
make_build_system.rst
drivers_design.rst
device_drivers.rst
processes_vs_tasks.rst
critical_sections.rst
interrupt_controls.rst
preemption_latency.rst
bottomhalf_interrupt.rst
cancellation_points.rst
chip_h.rst
context_switches.rst
critical_sections.rst
device_drivers.rst
device_nodes.rst
drivers_design.rst
file_descriptors.rst
hardfaults.rst
interrupt_controls.rst
ioctl.rst
kernel_modules_vs_shared_libraries.rst
kernel_threads_vs_pthreads.rst
make_build_system.rst
memory_configurations.rst
naming_arch_mcu_board_interfaces.rst
naming_os_internals.rst
nuttx_initialization_sequence.rst
nuttx_tasking.rst
oneshot_timers_and_cpu_load.rst
power_management.rst
preemption_latency.rst
processes_vs_tasks.rst
short_time_delays.rst
signal_handlers.rst
simulation.rst
smp.rst
syslog.rst
tasks_vs_threads.rst
tickless_os.rst
tls.rst
usb.rst

137
Documentation/implementation/ioctl.rst Normal file
View file

@ -0,0 +1,137 @@
.. _ioctl:
=====
ioctl
=====
Not a Typewriter
================
"In computing, **Not a typewriter** or ``ENOTTY`` is an error code defined
in the ``errno.h`` found on many Unix systems.
This code is now used to indicate that
an invalid ``ioctl`` (input/output control) number was specified
in an ``ioctl`` system call."
--Source: https://en.wikipedia.org/wiki/Not_a_typewriter.
NuttX
=====
In ``ioctl()`` implementations in NuttX, ``-ENOTTY`` is always returned
if the ``ioctl()`` command is not recognized. You will often see driver
``ioctl()`` implement ions with a general structure similar to the following:
.. code:: c
int driver_ioctl(FAR struct file *filep, int cmd, unsigned long arg)
{
int ret;
switch (cmd)
{
...
default:
ret = -ENOTTY;
break;
}
}
return ret;
}
Note that ``-ENOTTY`` is returned internally in NuttX.
This will subsequently be used to set the errno value to ``ENOTTY``
and to ``return -1`` to indicate the error condition.
ERRORS
------
These are the return values from a Linux ``ioctl()`` call:
* ``EBADF`` fd is not a valid file descriptor.
* ``EFAULT`` argp references an inaccessible memory area.
* ``EINVAL`` request or argp is not valid.
* ``ENOTTY`` fd is not associated with a character special device.
* ``ENOTTY`` The specified request does not apply to the kind of object
that the file descriptor fd references.
Reference: https://www.man7.org/linux/man-pages/man2/ioctl.2.html.
Linux Explanation
=================
On Jun 27 Linus Torvalds wrote:
"The correct error code for "I don't understand this ioctl" is ENOTTY.
The naming may be odd, but you should think of that error value as a
"unrecognized ioctl number, you're feeding me random numbers that I
don't understand and I assume for historical reasons that you tried to
do some tty operation on me".
...
The EINVAL thing goes way back, and is a disaster. It predates Linux
itself, as far as I can tell. You'll find lots of man-pages that have
this line in it:
EINVAL Request or argp is not valid.
and it shows up in POSIX etc. And sadly, it generally shows up
_before_ the line that says
ENOTTY The specified request does not apply to the kind of object
that the descriptor d references.
so a lot of people get to the EINVAL, and never even notice the ENOTTY.
(..)
At least glibc (and hopefully other C libraries) use a _string_ that
makes much more sense: strerror(ENOTTY) is "Inappropriate ioctl for
device"."
--Source: https://lore.kernel.org/patchwork/patch/258361.
How is this useful?
Knowing that no error occurred but the ``ioctl()`` command was not recognized
is a useful piece of information.
Suppose, for example, I have nfds open character drivers in an array ``fd[]``.
Then I could do something like this:
.. code:: c
int do_command(FAR int *fd, int nfds, int cmd)
{
int ret;
int i;
/* Try all file descriptors */
for (i = 0; i < nfds; i++)
{
ret = ioctl(fd[i], cmd, 0ul); /* No argument in this example */
if (ret < 0)
{
int errcode = errno;
/* Try the next file descriptor if this one return ENOTTY */
if (errcode != ENOTTY)
{
/* Other errors, including EINVAL, are fatal */
return -errcode
}
}
else if (ret >= 0)
{
return OK; /* Success! */
}
}
return -ENOENT;
}

176
Documentation/implementation/kernel_modules_vs_shared_libraries.rst Normal file
View file

@ -0,0 +1,176 @@
.. _kernel-modules:
==================================
Kernel Modules vs Shared Libraries
==================================
Kernel Modules
==============
NuttX has had support for Kernel Modules for some time.
A kernel module allows you to extend the functionality of the OS
at runtime by installing ELF modules into the kernel.
A kernel module might be used, to example, to load device drivers
into RAM at runtime.
Here are some general properties of kernel modules:
* The first kernel module is loaded into the kernel address space
by ``insmod()`` using only a symbol table exported by the OS.
* Kernel modules can also (optionally) export a symbol table.
Such symbol tables are remembered and will be used by ``insmod()``
to resolved undefined symbols in subsequently loaded modules.
* This requires dependency checking logic: A module that imports symbols
from another module must be added after the modules that it depends upon.
And a module exports a symbol may not be removed while there are such
inter-module dependencies in place.
A module that imports symbols from another module must be removed before
the module that exports the symbol can be removed.
* There is a (non-standard) OS interface ``modsym()`` that will allow kernel
logic to look up symbols within a kernel module.
* Handles are used at the kernel module interfaces: ``insmod()`` returns
a handle to the module data structure. That handle can subsequently be used
to retrieve symbols with ``modsym()``.
* ``rmmod()`` uses the handle to remove the module.
* ``modhandle()`` is also available for backward compatibility:
Given the assigned module name, you can use this to retrieve
the module handle at any time.
* There is a test case at ``nuttx-apps/examples/module``.
In the FLAT build, ELF kernel modules are simply loaded into RAM and linked
with the base firmware. But things get a little more complex with PROTECTED
and KERNEL builds.
In those case, there are separate address spaces for the kernel
and for applications.
Kernel modules are only loaded in the kernel address space and, hence,
are not available to applications.
Shared Libraries
================
A shared library is another software module with these properties:
The ``.text`` address space is accessible to all applications.
The ``.data`` and ``.bss`` address space is in the same address space
as the application that uses the shared library.
So, they are shared in the sense that the ``.text`` is shared.
FLAT Build
==========
In the FLAT build environment, there is only one address space.
So what is the difference between a kernel module and a shared library
in this case? Certainly a kernel module meets all of the requirements
of a shared library in that environment.
In this case kernel modules really only differ from shared
libraries in their usage semantics:
For the FLAT build, I have added the standard ``include/dllfcn.h``
and have implemented the FLAT shared library support as a thin wrapper
around the kernel module support:
* ``dlopen()`` maps to ``insmod()``.
* ``dlclose()`` maps to ``rmmod()``.
* ``dlsym()`` maps to ``modsym()``.
* ``dlerror()`` is only a stub at the present time.
There is a shared library test case at ``nuttx-apps/examples/sotest``.
PROTECTED Build
===============
The PROTECTED build is equivalent to the FLAT build except that there are
two address spaces: The kernel address space and the user address space.
But all applications still share the same user address spaces.
As a result ``.text`` along with ``.data`` and ``.bss`` are naturally shared.
This requires using two copies of the the module logic: One residing in kernel
address space and using the kernel symbol table and one residing in user space
using the user space symbol table.
The first provides only kernel module support; the second only PROTECTED mode
shared library support.
This is accomplished by breaking the kernel module logic in two components
with OS kernel module interfaces in sched/module but with a sharable module
library at libc/modlib.
The shared library functions no longer call the kernel module logic but rather
implement their one top-level management logic using the lower-level routines
in the module library.
Better FLAT and PROTECTED Mode Shared Libraries
===============================================
A better implementation of shared libraries in the FLAT and PROTECTED builds
would, however, have a separate copy of the ``.bss`` and ``.data`` region
for each NuttX task group.
A task group is the moral equivalent of a Unix process.
That is how a shared library would have to work in uClinux, for example.
But that would be a substantial effort! For example, since each
``.bss``/``.data`` would lie at a different physical addres,
the ``.text`` section logic would need support
Position-Independent-Data (PID).
Embedded PID support, however, is pretty much broken on all current GCC
implementations. See NxFlat Compatibility Problem.
Perhaps the xFLAT work that I did for uClinux shared libraries could be
ported to Nuttx: `xFLAT Web Page <https://sourceforge.net/projects/xflat/>`_.
For more information about task groups see :ref:`nuttx-tasking`
or :ref:`tasks-vs-threads`.
The current implementation also assumes that all firmware resides in base
FLASH and hence is fully linked prior to loading any modules or shared
libraries. There is no support for loading programs into RAM and binding
them to symbols exported by modules or shared libraries.
This extension would, however, not be so difficult for the FLAT build;
it would be a simple matter of integrating the exported module symbol
tables into the symbol lookup.
That is already done in ``sched/module`` files, but not yet in the
very similar ``binfmt/libelf`` files.
In the PROTECTED build, this would require some special start-up logic
in the user address space as the initial steps of the newly started task.
Some kind of dynamic loader, such as ``ld.so``, would have to integrate
with ``crt0`` logic to automatically bind user space tasks to shared libraries
as they are loaded into and memory before the programs ``main()`` function
is called.
KERNEL Build
============
The KERNEL build, however, is a completely different creature.
In that build, the kernel and each process has its own adress space.
This means that a shared library in the kernel build has to be considerably
more complex: In order to be shared, the ``.text`` portion of the module
(1) must lie in a single shared memory region accessible from all processes
and (2) built for Position-Independent-Code (PIC) operation since
it must execute from an arbitrarily different virtual address in each process
address space.
The ``.data``/``.bss`` portion of the module must be allocated
in the user address space of each process, but either:
1. ``.data``/``.bss`` section must lie at a consistent virtual address
so that it can be referenced from the one copy of the ``.text`` in
the shared memory region, OR
2. The ``.text`` section logic must support Position-Independent-Data (PID).
The latter approach provides for a simpler build, but embedded PID support
is pretty much broken on all current GCC implementations.
See :ref:`nxflat` Compatibility Problem.
Some kind of dynamic loader, such as ``ld.so``, would have to integrate with
``crt0`` logic to automatically bind processes to shared libraries as they
are loaded into memory before the programs ``main()`` logic is called.
.. note:: There is not yet any shared library support in the KERNEL build mode.
This would be quite a large effort and not on the plan of record
at the present time.

191
Documentation/implementation/kernel_threads_vs_pthreads.rst Normal file
View file

@ -0,0 +1,191 @@
.. _kernel-threads-vs-pthreads:
===========================
Kernel Threads vs. Pthreads
===========================
Why Can't Kernel Threads Have pthreads?
=======================================
Kernel threads are special "tasks" that reside within the OS.
They are similar to application tasks, so why can't they have pthreads
like application tasks?
The FLAT build
==============
In the FLAT build, kernel threads are, in fact, identical to application
threads except that:
1. They follow some slightly different syntax, AND
2. Can only use OS internal interfaces, never application interfaces.
Since they are otherwise identical, there is really no technical reason
why pthreads could not be supported.
The real reasons in this case is:
1. It is inappropriate, AND
2. It is incompatible with the PROTECTED and KERNEL builds where pthreads
cannot be supported.
Why inappropriate? Because..
pthread Interfaces Are User Interfaces
--------------------------------------
In all Unix systems, pthread interface support is not provided
by the operating system but rather by the user-facing C library.
The role of the OS is only to provide some low level hooks needed by
C library implementation of pthread interfaces.
But all of the implementation is in the C library and only for use
by applications.
That is not the current situation in NuttX.
Currently pthread support is deeply entangled in the OS.
This is problem that must be fixed someday and must not exploited as some
permanent feature of the OS. It is not.
It will go away some day when NuttX becomes a mature Unix-family system.
As user facing interfaces, pthread interfaces include some behaviors that
are not appropriate within the OS.
Inappropriate behaviors include modification of the ``errno`` value and
cancellation points. Those are user-only features.
While pthread interfaces do not, in general, modify the errno setting
they do create cancellation points which is not desirable within the OS.
PROTECTED and KERNEL Builds
---------------------------
The PROTECTED and KERNEL builds differ from the FLAT build in that they
segregate the memory into two regions:
A privileged kernel address space and unprivileged, user address space(s).
The primary difference is that the PROTECTED build uses a physical address
space and different regions of the physical address space have different
properties. This is usually accomplished using a Memory Protection Unit (MPU).
There is one protected kernel address space and one unprotected
user address space.
The KERNEL build, on the other hand, uses a Memory Management Unit (MMU)
to create a virtual address space in which there is still a separate
protected kernel address space, but many user address spaces for user
programs. These are usually called processes.
This is the familiar build model that you find with high-end Unix-like
systems such as Linux.
See Memory Configurations for additional information.
Address Spaces, Memory Allocators, and User Mode
------------------------------------------------
What would happen if you tried to create a pthread in the PROTECTED or KERNEL
build? The effect of these address space differences become very pronounced.
pthreads are, by their very definition, user-space threads.
That means that the OS will attempt to create a user-space environment
for the pthread: The thread's stack and other resources.
And when the pthread is started, it will be started in user mode,
not kernel mode.
It would require a significant change to the OS to alter that behavior
and that is not under consideration (because the change is also inappropriate).
Entry points
------------
Because the application space and the kernel space are separately built
and separately linked in the PROTECTED and KERNEL builds.
No application addresses are known by the kernel and no kernel addresses
are known by the application (applications interface via system call traps,
not C function calls). So what address would the kernel thread provide
for the pthread entry point? A known kernel space address?
Of course, the system would crash with a memory fault immediately
if a protected address were executed in user mode.
Mutexes
-------
Okay, so there are no kernel pthreads. But could other pthread resources
be usable in the kernel. The short answer is no.
Consider mutexes, for example. By definition, a mutex can only be used within
threads of the current process (or task groups as they are often called
in NuttX). They have no meaning outside of the task group.
Since the kernel thread "task group" can have no pthreads,
how could these mutexes be used under the proper standard definition
of what a mutex must be?
This is true of all pthread interfaces: None of the pthread interfaces were
intended from inter-process communications; only for inter-pthread
communications within the same process (task group).
Roadmap
=======
As alluded to before about the appropriateness of pthread in the OS.
There is a roadmap for these kinds of features.
That roadmap is to continue to conform strictly to the standard OS definitions
of <Opengroup.org>_ and to continue to evolve as a fully compliant,
very standard, tiny Unix-like operating system.
1. Part of this is getting all user interfaces out of the OS.
2. Another part is migrating all pthread support out of operating system
and into the user-facing C library.
3. An additional objective on the roadmap is to streamline the kernel threads
and to disconnect them from task groups.
Kernel threads should not be part of any task group.
In regard to this last point, the following is taken from Apache NuttX
Issue 1108: "Remove streams from Kernel Threads".
No Streams in Kernel Threads
----------------------------
Kernel threads are not permitted to use the C library buffer I/O, "stream",
interfaces. Those are interfaces like ``fopen()``, ``fread()``, ``fwrite()``,
``fclose()``, etc.
Those are strictly for use by user applications.
This is because these functions modify errno variables and create
cancellation points and perhaps other things that are undesirable
within the OS.
The thread create logic is largely the same for both kernel threads
and for application threads: They both allocate a large buffer
from the user memory pool for the stream ``FILE`` array.
Since kernel threads this is a waste of memory since the kernel threads
should not be using streams.
Remove Kernel Thread Stream Allocation
--------------------------------------
The proposal is to:
1. Verify that there is no use of streams within the OS,
2. Remove the stream allocation for kernel threads,
3. Assure that there are proper checks in place so that there are
no uses of the ``NULL`` stream array pointer.
For the most part, the OS is clean, there are essentially no use
of streams within the OS.
There are, however, a few violations of this that will need to be fixed;
``fopen()`` is called in some of the ``lc3850`` code.
Remove the File Descriptor Array Too
------------------------------------
A second phase would be to remove the file descriptor array from kernel
threads. File descriptors are, again, only for use by applications;
within the OS, file access is done using the struct file (aka, ``FILE``),
structure directly.
However, I suspect that there are many hidden uses of file descriptors
in the system.
For one, ``file_open()`` which opens the detached file, is not fully
implemented; it cheats and uses file descriptors.
So let's consider removal of the file descriptor allocation
as a second step after the stream allocations have been removed.

1000
Documentation/implementation/memory_configurations.rst Normal file
View file

File diff suppressed because it is too large Load diff

76
Documentation/implementation/naming_arch_mcu_board_interfaces.rst Normal file
View file

@ -0,0 +1,76 @@
=================================================
Naming of Architecture, MCU, and Board Interfaces
=================================================
What's the meaning of the prefix ``up_`` in NuttX?
Which functions should be prefixed with ``up_``?
``up_`` is supposed to stand for **microprocessor**;
the `u` is like the Greek letter micron: µ. So it would be **µP** which
is a common shortening of the word microprocessor.
I don't like that name very much. I wish I would have used ``arch_`` instead.
But now I think I am stuck with ``up_``.
Common Microcontroller Interfaces
=================================
Any interface that is common to all microcontroller and is used throughout
the system should be prefixed with ``up_`` and prototyped in
``include/nuttx/arch.h``. The definitions in that header file provide the
common interface between NuttX and the architecture-specific implementation
in ``arch/``.
Microcontroller-Specific Interfaces
===================================
An interface which is unique to a certain microcontroller should be prefixed
with the name of the microcontroller, for example ``stm32_``,
and be prototyped in some header file in the ``arch/`` directories.
These are interfaces used only by logic within the ``arch/`` and ``boards/``
directories.
Architecture-Specific Interfaces
================================
An interface which is unique to a CPU architecture, but shared among
microcontrollers that implement that CPU architecture should be prefixed
with the name of architecture. For example, the architecture-specific
interfaces used by STM32 would begin with ``arm_``.
These are interfaces used only by logic within the ``arch/`` and ``boards/``
directories.
There is also a ``arch/<architecture>/include/<chip>/chip.h`` header file
that can be used to communicate other microprocessor-specific information
between the board logic and even application logic.
Application logic may, for example, need to know specific capabilities
of the chip. Prototypes in that ``chip.h`` header file should follow
the microprocessor-specific naming convention.
Common Board Interfaces
=======================
Any interface that is common to all boards should be prefixed with ``board_``
and should also be prototyped in ``include/nuttx/arch.h``.
These ``board_`` definitions provide the interface between the board-level
logic and the architecture-specific logic.
There is also a ``boards/<arch>/<chip>/<board>/include/board.h`` header file
that can be used to communicate other board-specific information between
the architecture logic and even application logic.
Any definitions which are common between a single architecture and several
boards should go in this ``board.h`` header file;
``include/nuttx/arch.h`` is reserved for board-related definitions common
to all architectures.
Specific Interfaces
===================
Any interface which is unique to a board should be prefixed with the board
name, for example ``stm32f4discovery_``. Sometimes the board name is too long
so ``stm32_`` would be okay too. These should be prototyped in
``boards/<arch>/<chip>/<board>/src/<board>.h`` and should not be used outside
of that directory since board-specific definitions have no meaning outside
of the board directory.

181
Documentation/implementation/naming_os_internals.rst Normal file
View file

@ -0,0 +1,181 @@
===============================
Naming of OS Internal Functions
===============================
.. note:: Most of the naming examples used in this section no longer exist.
They have been converted to the naming conventions described
in this section. While they are good examples, they no longer
reflect the current state of the code base.
Legacy smooshed-together names
==============================
There has been some controvery lately about the consistency of naming of OS
internal functions. We should come to a consensus on the desired naming
of conventions for internal OS functions before we start arbitrarily
changing names.
NuttX function names are all lower case.
That is required by the naming standard.
Historically, has used a subsystem name, an underscore, then all of the
renaming words smooshed together. For example:
.. code-block:: c
void sched_mergeprioritized(FAR dq_queue_t *list1, FAR dq_queue_t *list2,
uint8_t task_state);
I would represent that form as::
<subsystem>_<everythingelsesmooshedaltogether>
Many of these names are impossible to read unless you stop and parse
the letters to find the word boundaries.
System-Object-Verb Naming
=========================
Most recent naming has broken out one more of the smooshed-together-words
and separated them with undersore characters like:
.. code-block:: c
unsigned int sched_timer_cancel(void);
void sched_timer_resume(void);
void sched_timer_reassess(void);
Which I would describe as::
<subsystem>_<object>_<verb>
System-Verb-Object Naming
=========================
Another form, which we have been using lately, is to switch the
``<object>`` and the ``<verb>`` like::
<subsystem>_<verb>_<object>
For example:
.. code-block:: c
int sched_get_stackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo);
By the ``<subsystem>_<object>_<verb>`` naming this would have been:
.. code-block:: c
int sched_stackinfo_get(pid_t pid, FAR struct stackinfo_s *stackinfo);
And the timer interfaces would become the following under the
``<subsystem>_<verb>_<object>`` rule:
.. code-block:: c
unsigned int sched_cancel_timerl(void);
void sched_resume_timer(void);
void sched_reassess_timer(void);
And the "smooshed" name ``sched_mergeprioritized()`` would become:
.. code-block:: c
void sched_merge_prioritized(FAR dq_queue_t *list1, FAR dq_queue_t *list2,
uint8_t task_state);
We won't even consider the ``<subsystem>_<everythingelsesmooshedaltogether>``.
It has a history but is pretty much odious.
We should not consider, for example, the smooshed-together form:
.. code-block:: c
int sched_getstackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo);
What an ugly mess! :-)
Unit Suffixes
=============
Often, you will have the functions that return the same value,
but in different units.
For example, these two functions return the system time:
.. code-block:: c
clock_t clock_systimer(void);
int clock_systimespec(FAR struct timespec *ts);
The first returns the time in system clock ticks.
The second returns the time as a struct timespec.
I have seen a naming practice The keeps the naming of the functions the same,
but appends the units, preceded by an underscore character, at the end
of the function name. For example, these could be:
.. code-block:: c
clock_t clock_systime_ticks(void);
int clock_systime_timespec(FAR struct timespec *ts);
Or should they include a ``<verb>``:
.. code-block:: c
clock_t clock_get_systime_ticks(void);
int clock_get_systime_timespec(FAR struct timespec *ts);
That form would then be::
<subsystem>_<verb>_<object>[_<units>]
Where the square bracket represent and optional unit suffix.
Implicit get
============
In many cases, I think that the ``_get_`` is implicit and need not appear
in the function name. But if there are multiple operations
on the ``<object>``, then the ``_get_`` would be necessary.
Hence, I think the naming:
.. code-block:: c
clock_t clock_systime_ticks(void);
int clock_systime_timespec(FAR struct timespec *ts);
is perfectly accepable without the ``_get_``.
And since ``_get_`` is the only operation on ``stackinfo``,
the following would be a perfectly acceptable renaming:
.. code-block:: c
int sched_stackinfo(pid_t pid, FAR struct stackinfo_s *stackinfo);
Other Naming
============
There is other naming that uses long names with many underscore characters
separating each word in the long name. We are not a fan of long names.
Conclusion
==========
Our preference would be the ``<subsystem>_<verb>_<object>`` form.
We have been using this form in creating all new internal interfaces.
Does anyone else have a thought? or a preference?
We should come to an understanding and stop the arbitrary name changes.
We hope that we can put an end to the smooshed-together naming
(except wheree required) in any case.
.. note:: This applies only to the internal naming of functions within the OS.
Other names are imposed on us by standards for application
interfaces to the OS which may follow other rules.

713
Documentation/implementation/nuttx_initialization_sequence.rst Normal file
View file

@ -0,0 +1,713 @@
.. _nuttx-initialization-sequence:
=============================
NuttX Initialization Sequence
=============================
Overview
========
This initialization sequence is really quite simple because the system runs
in single-thread mode up until the point the that is starts the application.
That means the initialization sequence is just a simple, straight-line
of function calls.
It is until just before starting the application that system goes to
multi-threaded mode and things can get more complex.
At the highest level, the NuttX initialization sequence can be
represented in three phases:
* **Phase A** - The hardware-specific power-on reset initialization,
* **Phase B** - NuttX RTOS initialization, and
* **Phase C** - Application Initialization.
Each of these will be discussed in more detail in the following sections.
Case example: STM32 F4
======================
In this discussion, we'll use the STM32 F4 MCU and its popular evaluation
board STM32F4Discovery as example but the explanation can be applied
to any supported architecture.
Here is the map of initialization function calls::
__start()-arch/arm/src/stm32/stm32_start.c
|
+--*Set stack limit
+--stm32_clockconfig()
+--stm32_fpuconfig()
+--stm32_lowsetup()
+--stm32_gpioinit()
+--showprogress('A')
+--
+--
+--stm32_boardinitialize()-boards/arm/stm32/stm32f4discovery/src/stm32_boot.c
| |
| +--stm32_spidev_initialize()-stm32_spi.c:ONLY CHIP SELECTS
| +--stm32_usbinitialize()-
| +--stm32_netinitialize()-
| +--board_autoled_initialize()-
|
nx_start()-sched/init/nx_start.c
|
+--*Initialize global data structures
+--*Initialize OS facilities
+--net_initialize()-net/net_initialize.c
| |
| +--net_lockinitialize()
| +--mld_initialize()
| +--can_initialize()
| +--netlink_initialize()
| +--tcp_initialize()
| +--udp_initialize()
| +--usrsock_initialize()
|
+--up_initialize()-arch/arm/src/common/up_initialize.c
| |
| +--arm_dmainitialize()
| +--Config basic /dev nodes
| +--arm_serialinit()
| +--Console Init
| +--Crypto Config
| +--arm_netinitialize()
| | |
| | +--stm32_spibus_initialize()
| |
| |
| +--arm_usbinitialize()
| +--L2 Cache Init
|
+--board_early_initialize()
+--g_nx_initstate = OSINIT_HARDWARE
+--shm_initialize()
+--lib_initialize()
+--binfmt_initialize()
+--Start SMP
+--syslog_initialize()
+--g_nx_initstate = OSINIT_OSREADY
+--DEBUGVERIFY(nx_bringup())-sched/init/nx_bringup.c
| |
| +--nx_pgworker()
| +--nx_workqueues()
| +--nx_create_initthread()-sched/init/nx_bringup.c
| |
| +----+different thread
| : :
| : nx_start_task()-sched/init/nx_bringup.c
| : :
| same+----+--nx_start_application()-sched/init/nx_bringup.c
| thread |
| +--board_late_initialize()-stm32_boot.c:BOARD_DEPENDANT
| | |
| | +--stm32_bringup()
| | |
| | +--stm32_i2ctool()
| | +--board_bmp180_initialize()
| | +--stm32_sdio_initialize()
| | +--stm32_usbhost_initialize()
| | +--stm32_pwm_setup()
| | +--stm32_can_setup()
| | +--etc
| |
| +--
|
|
|
|
|
+--kmm_givesemaphore()
+--up_idle()
Phase A - Power-On Reset Initialization
=======================================
The system begins execution when the processor is reset.
This usually at power-on, but all resets are basically the same whether
they occur because of power-on, pressing the reset button, or on a watchdog
timer expiration.
.. note:: The code that executes when the processor is reset is unique
to the particular CPU architecture and is not a common part
of NuttX.
The kind of things that must be done by the architecture-specific reset
handling includes:
* Putting the processor in its operational state. This may include things
like setting CPU modes; initializing co-processors, etc.
* Setting up clocking so that the software and peripherals operate as expected.
* Setting up the C stack pointer (and other processor registers).
* Initializing memory.
* Starting NuttX.
Memory Initialization
---------------------
In C implementations, there are two general classes of variable storage.
First there are the initialized variables.
For example, consider the global variable x:
.. code-block:: c
int x = 5;
The C code must be assured that after reset, the variable x has the value 5.
Initialized variable of this kind are retained in a special memory section
called data (or ``.data``).
Other variables are not initialized. Like the global variable y:
.. code-block:: c
int y;
But the C code will still expect y to have an initial value.
That initial value will be zero.
All uninitialized variables of this type need to have the value zero.
These uninitialized variables are retained in a section called bss
(or ``.bss``).
When we say that the reset handling logic initializes memory, we mean two things:
1. It provides the (initial) values of the initialized variables by copying
the values from FLASH into the ``.data`` section, and
2. It resets all of the uninitialized variables to zero.
It clears the ``.bss`` section.
Lets walk through reset sequence. This reset logic can be found in two files::
nuttx/arch/arm/src/stm32_vectors.S
nuttx/arch/arm/src/stm32_start.c
nuttx/arch/arm/src/stm32_vectors.S
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This file provides all of the STM32 exception vectors and power-on reset
is simply another exception vector.
There are few important things to note about this file.
``.section .vectors, ax``. This pseudo operation will place all of the
vectors into a special section call ``.vectors``.
On of the STM32 F4 linker scripts is located at
``nuttx/boards/arm/stm32/stm32f4discovery/scripts/ld.script``.
In that file, you can see that section ``.vectors`` is forced to lie
at the very beginning of FLASH memory.
The STM32 F4 can be configured to boot in different ways via strapping.
If it is strapped to boot from FLASH, then the STM32 FLASH memory will
be aliased to address ``0x0000 0000`` when the reset occurs.
That is the address of the power-up reset interrupt vector.
The first two 32-bit entries in the vector table represent the power-up
exception vector (which we know will be positioned at address
``0x0000 0000`` when the reset occurs). Those two entries are:
.. code-block:: c
.word IDLE_STACK /* Vector 0: Reset stack pointer */
.word __start /* Vector 1: Reset vector */
The Cortex-M family is unique in the way that is handles the reset vector.
Notice that there are two values: the stack pointer for the start-up thread
(the IDLE thread), and the entry point in the IDLE thread.
When the reset occurs, the the stack pointer is automatically set
to the first value and then the processor jumps to reset entry point
``__start`` specified in the second entry.
This means that the reset exception handling code can be implemented in C
rather than assembly language.
nuttx/arch/arm/src/stm32_start.c
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The reset vector ``__start`` lies in the file
``nuttx/arch/arm/src/stm32/stm32_start.c`` and does the real,
low-level architecture-specific initialization. This initialization includes:
1. ``stm32_clockconfig()`` - Initialize the PLLs and peripheral clocking
needed by the board.
2. ``stm32_fpuconfig()`` - If the STM32 F4's hardware floating point
is initialized, then configure the FPU and enable access to the FPU
co-processors.
3. ``stm32_lowsetup()`` - Enable the low-level UART. This is done very early
in initialization so that we can get serial debug output to the console
as soon as possible. If you are doing a board bring-up this
is very important.
4. ``stm32_gpioinit()`` - Perform any GPIO remapping that is needed
(this is a stub for the F4, but the F1 family requires this step).
5. ``showprogress('A')`` - This simply outputs the character ``A`` on the
serial console (only if ``CONFIG_DEBUG`` is enabled). If debug is enabled,
you will always see the letters ABDE output on the console. That output
all comes from this file.
6. Next the memory is initialized:
a. The ``.bss`` section is set to zero (Letter ``B`` is then output if
``CONFIG_DEBUG`` is enabled), then
b. The ``.data`` section is set to its initial values (The letter ``C``
is output if debug is enabled),
7. ``stm32_boardinitialize()`` - Board-specific logic is initialized by
calling this function. For the case of the STM32F4Discovery board,
this logic can be found at
``nuttx/boards/arm/stm32/stm32f4discovery/src/stm32_boot.c`` and does
the following operations:
a. ``stm32_spidev_initialize()`` - Initialize SPI chip selects
if SPI is enabled.
b. ``stm32_usbinitialize()`` - Initialize hardware USB devices if enabled.
c. ``stm32_netinitialize()`` - Initialize hardware network devices
if enabled.
d. ``board_autoled_initialize()`` - Configure on-board LEDs
if LED support has been selected.
8. When ``stm32_boardinitialize()`` returns to ``__start()``, the low-level,
architecture-specific initialization is complete.
Phase B - NuttX RTOS Initialization
===================================
``nx_start()``
--------------
This function resides in the file ``nuttx/sched/init/nx_start.c`` and
is the NuttX entry point.
It is called by ``__start()`` and performs the next phase of RTOS-specific
initialization before bringing up the application.
The operations performed by ``nx_start()`` are summarized below.
Note that many of these features can be disabled from the NuttX configuration
file and in that case those operations are not performed:
1. Initializes some NuttX global data structures,
2. Initializes the TCB for the IDLE (i.e, the thread that the initialization
is performed on),
3. ``nxsem_initialize()`` - Initialize the POSIX semaphore facilities.
This needs to be done first because almost all other OS features depend
on POSIX counting semaphores.
4. Memory organization - This includes heap configuration, memory manager,
paging, I/O buffers, etc.
5. ``task_initialize()`` - Initialize task data structures.
6. ``fs_initialize()`` - Initialize the file system (needed to support
device drivers).
7. ``irq_initialize()`` - Initialize the interrupt handler subsystem.
This initializes only data structures; CPU interrupts are still disabled.
8. ``wd_initialize()`` - Initialize the NuttX watchdog timer facility,
9. ``clock_initialize()`` - Initialize the system clock,
10. ``timer_initialize()`` - Initialize the POSIX timer facilities,
11. ``nxsig_initialize()`` - Initialize the POSIX signal facilities,
12. ``nxmq_initialize()`` - Initialize the POSIX message queue facilities,
13. ``pthread_initialize()`` - Initialize the POSIX pthread facilities,
14. ``net_initialize()`` - Initialize networking facilities,
.. note:: Up to this point, all of the initialization steps have only been
software initializations. Nothing has interacted with the hardware.
Rather, all of these steps simply prepared the environment so that
things like interrupts and threads can function properly.
The next phases depend upon that setup.
15. ``up_initialize()`` - The processor specific details of running
the operating system will be handled here. Such things as setting up
interrupt service routines and starting the clock are some of the things
that are different for each processor and hardware platform.
All ARM-based MCUs share a common ``up_initialize()`` implementation
provided at ``nuttx/arch/arm/src/common/up_initialize.c``. The operations
perform by this common ARM initialization will, however, call into
facilities provided by the particular ARM chip.
For the STM32 F4, those facilities would be provided by logic in files
as ``nuttx/arch/arm/src/stm32``. The common ARM initialization sequence is:
* ``up_color_intstack()`` - Colorize the interrupt stack.
* ``arm_addregion()`` - The basic heap was set up during processing by
``nx_start()``. However, if the board supports multiple, discontiguous
memory regions, any addition memory regions can be added to the heap
by this function. For the STM32 F4, ``up_addregion()`` is implemented
in ``nuttx/arch/arm/src/stm32/stm32_allocateheap.c``.
* ``arm_pminitialize()`` - If ``CONFIG_PM`` is defined, the function must
initialize the power management subsystem. This MCU-specific function
must be called very early in the intialization sequence before any other
device drivers are initialized (since they may attempt to register with
the power management subsystem). There is no implementation
of ``up_pminitialize()`` for any STM32 platform.
* ``arm_dmainitialize()`` - Initialize the DMA subsystem.
For the STM32 F4, this DMA initialization can be found in
``nuttx/arch/arm/src/stm32/stm32_dma.c`` (which includes
``nuttx/arch/arm/src/stm32f4xxx_dma.c``).
* ``devnull_register()`` - Registers the standard ``/dev/null``.
* ``devrandom_register()`` - Registers the standard ``/dev/random``.
* ``devurandom_register()`` - Registers the standard ``/dev/urandom``.
* ``devzero_register()`` - Registers the standard ``/dev/zero``.
* ``loop_register()`` - Registers the standard ``/dev/loop``.
* ``note_register()`` - Registers the standard ``/dev/note``.
* ``arm_serialinit()`` - Initialize the **standard** serial driver
(found at ``nuttx/arch/arm/src/stm32/stm32_serial.c`` STM32 F4).
* ``arm_netinitialize()`` - Initialize the network. For the STM32 F4,
this function is in ``nuttx/arch/arm/src/stm32/stm32_eth.c``.
* ``arm_usbinitialize()`` - Initialize USB (host or device).
For the STM32 F4, this function is in
``nuttx/arch/arm/src/stm32/stm32_otgfsdev.c``.
* ``arm_l2ccinitialize()`` - Initialize the L2 cache if present
and selected.
* ``up_ledon(LED_IRQSENABLED)`` - Finally, ``up_initialize()``
illuminates board-specific LEDs to indicate the IRQs are now enabled.
16. ``board_early_initialize()`` - If ``CONFIG_BOARD_EARLY_INITIALIZE`` is
selected, then an additional initialization call will be performed in the
boot-up sequence to a function called ``board_early_initialize()``.
It will be called immediately after ``up_initialize()`` (and may be
thought of as a board-specific, extension of ``up_initialize()``)
and well ``before board_late_initialize()`` is called and the initial
application is started.
17. ``g_nx_initstate = OSINIT_HARDWARE`` - This signals that basic
hardware setup is complete.
18. ``shm_initialize()`` - Initialize shared memory support.
19. ``lib_initialize()`` - Initialize the C libraries. This is done last
because the libraries may depend on the above.
20. ``binfmt_initialize()`` - Initialize binary loader subsystem.
21. Start SMP support in multi-core MCUs. This is not the case
in STM32F4Discovery but relevant because here are created ``stdin``,
``stdout`` and ``stderr`` for each CPU's (even if there is only one)
IDLE task. All tasks subsequently created by the IDLE thread will inherit
these file descriptors.
22. ``syslog_initialize()`` - Late initialization of the system logging
device. Some SYSLOG channel must be initialized late in the initialization
sequence because it may depend on having IDLE task file structures setup.
23. ``nx_bringup()`` - Create the initial tasks. This will be described
in more detail below.
``nx_bringup()``
----------------
This function is called at the very end of the initialization sequence
in ``nx_start()``, just before entering the IDLE loop. It is located
in ``nuttx/sched/init/nx_bringup.c`` and it starts all of the required
threads and tasks needed to bring up the system.
This function performed the following specific operations:
* ``nx_pgworker()`` - Start the page fill worker kernel thread that will
resolve page faults. This should always be the first thread started because
it may have to resolve page faults in other threads. This is the task that
runs in order to satisfy page faults in processors that have an MMU and
in configurations where on-demand paging is enabled.
* ``nx_workqueues()`` - Start the worker thread. The worker thread may be used
to execute any processing deferred to the worker thread via APIs provided
in ``include/nuttx/wqueue.h``. The worker thread's primary function is
as the “bottom half” for extended device driver processing but can be used
for a variety of purposes like misc garbage clean-up.
* ``nx_create_initthread()`` - Once the operating system has been initialized,
this funcions either directly calls ``nx_start_application()`` or creates
a thread for running it
* ``nx_start_application()`` - If set in the NuttX configuration,
this function calls ``board_late_initialize()``.
* ``board_late_initialize()`` is a last-minute, board-specific
initialization. Note that there was earlier, board-specific initialization
calls (to ``stm32_board_initialize()`` and
to ``board_early_initialize()``). The difference here is these first,
low-level initialization calls were made before the OS was completely
launched. ``board_late_initialize()``, on the other hand, is called
at the end after the OS has been initialized but before any application
tasks have been started.
``board_late_initialize()`` would be an ideal place to do board-specific
initialization steps that depend on having a fully initialized OS
such as memory allocations, initialization of complex device drivers,
mounting of file systems, etc.
* After that, ``nx_start_application()`` launches the application either
by creating a task for it or executing a program from a filesystem after
mounting it.
* In the case of creating a task for the application, its entry has
the name ``user_start()``. ``user_start()`` is provided by application
code and when it runs, it begins the application-specific phase of the
initialization sequence as described below.
.. note:: The default ``user_start()`` entry point can be changed to use
one of the named applications used by NSH. This is a start-up option
that is not often used and will not be discussed further here.
And finally enter the IDLE loop. After completing the initialization, the role
of the IDLE thread changes. It becomes the thread that executes only when
there is nothing else to do in the system (hence, the name IDLE thread).
IDLE Thread Activities
----------------------
As mention, the IDLE thread is the thread that executes only when there is
nothing else to do in the system. It has the lowest priority in the system.
It always has the priority 0. It is the only thread that is permitted to have
the priority 0. And it can never be blocked (otherwise, what would run then?).
As a result, the IDLE thread is always in the ``g_readytorun`` list and,
in fact, since that list is prioritized, can guaranteed to always be
the final entry at the tail of the ``g_readytorun`` list.
The IDLE is an an infinite loop. But this does not make it a “CPU hog”.
Since it is the lowest priority, the it can be suspended whenever anything
else needs to run.
The IDLE thread does two things in this infinite loop:
1. If the worker was not started (see ``nx_bringup()`` below), then the IDLE
thread will perform memory clean-up. Memory clean is required to handle
deferred memory deallocation. Memory allocations must be deferred when
the memory is freed in a context where the software does not have access
to the heap and, hence, cannot truly free the memory (such as
in an interrupt handler). In this case, the memory is simply put into
a list of freed memory and, eventually, cleaned up by the IDLE thread.
.. note:: The worker thread's primary function is as the “bottom half”
for extended device driver processing. If the worker thread
was started, then it will run at a higher priority than the
IDLE thread. In this case, the worker thread will take over
responsibility for cleaning up these deferred allocations.
2. Then the loop calls ``up_idle()``. The operations performed
by ``up_idle()`` are architecture- and board-specific. In general,
this is the location where CPU-specific reduced power operations
may be performed.
STM32 F4 IDLE thread
--------------------
The default STM32 F4 IDLE thread is located at
``nuttx/arch/arm/src/stm32_idle.c``.
This default version does very little:
1. It includes a example, “skeleton” function that illustrates that kinds
of things that you can do if ``CONFIG_PM`` is enabled (this example code
is not fully implemented in the default IDLE logic).
2. The it executes the Cortex-M thumb2 instruction wfi which causes the CPU
to sleep until the next interrupt occurs.
Phase C - Application Initialization
====================================
At the conclusion of the OS initialization phase in ``nx_start()``,
the user application is started by creating a new task
at the entry point ``user_start()``.
There must be exactly one entry point called ``user_start()``
in every application built on top of NuttX.
Any additional initialization performed in the ``user_start()`` function
is purely application dependent.
A Simple Hello World Application
--------------------------------
The simplest user application would be the “Hello, World!” example.
See ``apps/examples/hello``. Here is the whole example:
.. code-block:: c
int user_start(int argc, char *argv[])
{
printf("Hello, World!!\n");
return 0;
}
In this case, no additional application initialization is needed.
It just “says hello” and exits.
A Nutt Shell User Application/Command
-------------------------------------
The NuttShell (NSH) is a simple shell application that may be used with NuttX.
It is described here. It supports a variety of commands and is (very) loosely
based on the bash shell and the common utilities used
in Unix shell programming.
NSH is implemented as a library that can be found at ``nuttx-apps/nshlib``.
The NSH start-up sequence is very simple. As an example, the the code at
``apps/examples/nsh/nsh_main.c`` illustrates how to start NuttX.
It simple does the following:
1. If you have C++ static initializers, it will call your implementation
of ``up_cxxinitialize()`` which will, in turn, call those
static initializers.
2. This function then calls ``nsh_initialize()`` which initializes
the NSH library. ``nsh_initialize()`` is described in more detail below.
3. If the Telnet console is enabled, it calls ``nsh_telnetstart()`` which
resides in the NSH library. ``nsh_telnetstart()`` will start the Telnet
daemon that will listen for Telnet connections and start remote
NSH sessions.
4. If a local console is enabled (probably on a serial port), then
``nsh_consolemain()`` is called. ``nsh_consolemain()`` also resides
in the NSH library. ``nsh_consolemain()`` does not return so that
finished the entire NSH initialization sequence.
``nsh_initialize()``
^^^^^^^^^^^^^^^^^^^^
The NSH initialization function, ``nsh_initialize()``, be found in
``apps/nshlib/nsh_init.c``. It does only three things:
``nsh_romfsetc()`` if so configured, it executes an NSH start-up script that
can be found at ``/etc/init.d/rcS`` in the target file system.
``/etc`` is the location where a read-only, ROMFS file system is mounted by
``nsh_romfsetc()``. The ROMFS image is, itself, just built into the firmware.
By default, this rcS startup script contains the following logic:
.. code-block:: sh
# Create a RAMDISK and mount it at XXXRDMOUNTPOUNTXXX
mkrd -m XXXMKRDMINORXXX -s XXMKRDSECTORSIZEXXX XXMKRDBLOCKSXXX
mkfatfs /dev/ramXXXMKRDMINORXXX
mount -t vfat /dev/ramXXXMKRDMINORXXX XXXRDMOUNTPOUNTXXX
Where the ``XXXX*XXXX`` strings get replaced in the template
when the ROMFS image is created:
* ``XXXMKRDMINORXXX`` will become the RAM device minor number. Default: ``0``.
* ``XXMKRDSECTORSIZEXXX`` will become the RAM device sector size.
* ``XXMKRDBLOCKSXXX`` will become the number of sectors in the device.
* ``XXXRDMOUNTPOUNTXXX`` will become the configured mount point.
Default: ``/etc``.
This script will, then, create a RAMDISK, format a FAT file system
on the RAM disk, and then mount the FAT filesystem at a configured mountpoint.
This rcS template file can be found at ``apps/nshlib/rcS.template``.
The resulting ROMFS file system can be found in
``apps/nshlib/nsh_romfsimg.h``.
* ``boardctl()`` - Next any architecture-specific NSH initialization will be
performed (if any). The NSH initialization logic will all the non-standard
OS interface ``boardctl()`` like: ``(void)boardctl(BOARDIOC_INIT, 0);``.
The first argument, the command ``BOARDIOC_INIT``, indicates that the
``boardctl()`` is being requested to perform application-oriented
initialization. In response to this command, ``boardctl()`` will call the
board-specific implementation of ``board_app_initialize()``.
That function is not generally available to application level code in
all configurations but can always be accesses via ``boardctl()``.
* ``board_app_initialize()`` - For the STM32F4Discovery, this architecture
specific initialization can be found at
``boards/arm/stm32/stm32f4discovery/src/stm32_appinit.c``.
This it does things like:
1. Initialize SPI devices.
2. Initialize SDIO.
3. Mount any SD cards that may be inserted.
There are two possibilities then for application startup initialization:
1. In the application itself via ``boardctl(BOARDIOC_INIT, 0)``, OR
2. in the OS using ``board_late_initialize()``.
Each works well in certain contexts but there are also things that I don't
like about either possibility.
``boardctl()``
--------------
You can see this application-controlled initialization in, for example,
the NSH application in `nuttx-apps/nshlib``. This is seen as the call to
``boardctl(BOARDIOC_INIT, 0)``. This requires that the ``boardctl()``
interface be enabled with ``CONFIG_LIB_BOARDCTL``.
When ``CONFIG_LIB_BOARDCTL=y``, board-specific OS internal logic must provide
the interface ``board_app_initilize()``. ``boardctl(BOARDIOC_INIT, 0)`` is the
application level interface that provides the proper interface to
``board_app_initilize()``. All board-specific, driver-level initialization may
then be performed in ``board_app_initilize()`` under the control
of the application.
With ``CONFIG_LIB_BOARDCTL=y``, there are two levels of application-specific
initialization:
(a) an OS/kernel level application initialization that needs to be performed
before the application is started, and
(b) a user/application level initialization that can be performed after
the application has started.
``board_late_nitialize()``
--------------------------
``board_late_initialize()`` is the OS/kernel level application initialization
that is performed before the application has started. It usually works
quite well and is usually a good alternative for ``boardctl(BOARDIOC_INIT)``.
To avoid issues with trying to perform initialization on the IDLE thread,
``board_late_initialize()`` is called from an internal kernel thread.
The IDLE thread has limitations: It cannot wait for events so
it can only be used for simple, straight-line initialization logic.
That may be insufficient in some cases.
For this reason, ``board_late_initialize()`` must run on a kernel thread.
Consider the ``board_late_initialize()`` calling sequence:
``board_late_initialize()`` is called directly from ``do_app_start()`` in
``nuttx/sched/init/nx_bringup.c`` just before starting the application task.
``nx_create_initthread()`` is, in turn, called from the function
``nx_start_application()``. If ``CONFIG_BOARD_LATE_INITIALIZE`` is
not defined, then this is just a normal C function call.
But if ``CONFIG_BOARD_LATE_INITIALIZE`` is defined, then an intermediate,
trampoline kernel thread is started. That kernel thread executes
``do_app_start()`` and moves the initialization off of the IDLE thread.
This works great but is a little more complex than I would like.
``apps/platform``
-----------------
There is also special place for user/application initialization.
That is the ``nuttx-apps/platform`` platform directory. This directory
should be a mirror of the ``nuttx/configs`` directory.
There should be a board directory in ``nuttx-apps/platform`` for every
board that is in ``nuttx/configs``.

380
Documentation/implementation/nuttx_tasking.rst Normal file
View file

@ -0,0 +1,380 @@
.. _nuttx-tasking:
=============
NuttX Tasking
=============
An RTOS as a library
====================
What is an RTOS? NuttX, as with all RTOSs, is a collection of various features
bundled as a library. It does not execute except when either:
1. The application calls into the NuttX library code, OR
2. An interrupt occurs.
There is no meaningful way to represent an architecture that is implemented
as a library of user managed functions with a diagram.
You can however, pick any subsystem of an RTOS and represent
that in some fashion.
Kernel Threads
==============
There are some RTOS functions that are implemented by internal threads,
for instance :ref:`kernel-threads-vs-pthreads`, :ref:`tasks-vs-threads`,
:ref:`kernel-modules`.
.. todo:: Provide more content here :-)
The Scheduler
=============
Schedulers and Operating Systems
--------------------------------
An operating system is a complete environment for developing applications.
One important component of an operating system is the scheduler.
That logic that controls when tasks or threads execute.
Actually, more than that; the scheduler really determines what a task
or a thread is! Most tiny operating systems are really not operating
“systems” in the sense of providing a complete operating environment.
Rather these tiny operating systems consist really only of a scheduler.
That is how important the scheduler is.
Task Control Block (TCB)
------------------------
In NuttX a thread is any controllable sequence of instruction execution
that has its own stack.
Each task is represented by a data structure called a task control block
or TCB. That data structure is defined in the header file
``include/nuttx/sched.h``.
Task Lists
----------
These TCBs are retained in lists. The state of a task is indicated both
by the ``task_state`` field of the TCB and by a series of task lists.
Although it is not always necessary, most of these lists are prioritized
so that common list handling logic can be used (only the ``g_readytorun``,
the ``g_pendingtasks``, and the ``g_waitingforsemaphore`` lists
need to be prioritized).
All new tasks start in an initial, non-running state:
.. code-block:: c
volatile dq_queue_t g_inactivetasks;
* This is the list of all tasks that have been initialized, but not yet
activated. NOTE: This is the only list that is not prioritized.
* When the task is initialized, it is moved to a ready-to-run list.
There are two lists representing ready-to-run threads and several
lists representing blocked threads. Here are the ready-to-run threads:
.. code-block:: c
volatile dq_queue_t g_readytorun;
* This is the list of all tasks that are ready to run.
The head of this list is the currently active task;
the tail of this list is always the idle task.
.. code-block:: c
volatile dq_queue_t g_pendingtasks;
* This is the list of all tasks that are ready-to-run, but cannot be placed
in the ``g_readytorun`` list because:
1. They are higher priority than the currently active task at the head
of the ``g_readytorun`` list, AND
2. the currently active task has disabled pre-emption.
These tasks will stay in this holding list until pre-emption is again
enabled (or until the currently active task voluntarily relinquishes
the CPU).
* Tasks in the ``g_readytorun`` list may become blocked.
In this cased, their TCB will be moved to one of the blocked lists.
When the block task is ready-to-run, its TCB will be moved back to either
the ``g_readytorun`` to ``the g_pendingtasks`` lists, depending up
if pre-emption is disabled and upon the priority of the tasks.
Here are the blocked task lists:
.. code-block:: c
volatile dq_queue_t g_waitingforsemaphore;
* This is the list of all tasks that are blocked waiting for a semaphore.
.. code-block:: c
volatile dq_queue_t g_waitingforsignal;
* This is the list of all tasks that are blocked waiting for a signal
(only if signal support has not been disabled).
.. code-block:: c
volatile dq_queue_t g_waitingformqnotempty;
* This is the list of all tasks that are blocked waiting for a message queue
to become non-empty (only if message queue support has not been disabled).
.. code-block:: c
volatile dq_queue_t g_waitingformqnotfull;
* This is the list of all tasks that are blocked waiting for a message queue
to become non-full (only if message queue support has not been disabled).
.. code-block:: c
volatile dq_queue_t g_waitingforfill;
* This is the list of all tasks that are blocking waiting for a page fill
(only if on-demand paging is selected).
Reference: ``nuttx/sched/sched/sched.h``.
State Transition Diagram
========================
The state of a thread can then be easily represented with this simple state
transition diagram.
.. todo:: Provide State Transition Diagram.
Scheduling Policies
===================
In order to be a real-time OS, an RTOS must support ``SCHED_FIFO``.
That is, strict priority scheduling. The thread with the highest priority
runs.. Period. The thread with the highest priority is always associated
with the TCB at the head of the ``g_readytorun`` list.
NuttX supports one additional real-time scheduling policy: ``SCHED_RR``.
The RR stands for **round-robin** and this is sometimes called
**round-robin scheduling**. In this case, NuttX supports timeslicing.
If a task with ``SCHED_RR`` scheduling policy is running, then when each
timeslice elapses, it will give up the CPU to the next task that is
at the same priority.
.. note::
1. If there is only one task at this priority, ``SCHED_RR`` and
``SCHED_FIFO`` are the same, AND
2. ``SCHED_FIFO`` tasks are never pre-empted in this way.
Task IDs
========
Each task is represented not only by a TCB but also by a numeric task ID.
Given a task ID, the RTOS can find the TCB.
Given a TCB, the RTOS can find the task ID.
So they are functionally equivalent.
Only the task ID, however, is exposed at the RTOS/application interfaces.
NuttX Tasks
===========
Processes vs. Threads
---------------------
In larger system OS such as BSD, Linux, or Windows you will often hear
the name process used to refer to threads managed by the OS.
A process is more than a thread as we have been discussing so far.
A process is a protected environment that hosts one or more threads.
By environment we mean the set of resources set aside by the OS but
in the case of the protected environment of the process we are specifically
referring its address space.
.. note::
In order to implement the process' address space, the CPU must support
a memory management unit (MMU).
**The MMU is used to enforce the protected process environment.**
However, NuttX was designed to support the more resource constrained,
lower-end, deeply embedded MCUs. Those MCUs seldom have an MMU and,
as a consequence, can never support processes as are supported by BSD, Linux,
or Windows.
.. important:: NuttX does not support processes.
NuttX will support an MMU but it will not use the MMU to support processes.
NuttX operates only in a flat address space.
NuttX will use the MMU to control the instruction and data caches and
to support protected memory regions.
This may change in future, but this is how things are right now.
NuttX Tasks and Task Resources
------------------------------
All RTOSs support the notion of a task. A task is the RTOS's moral equivalent
of a process. Like a process, a task is a thread with an environment
associated with it.
This environment is like environment of the process but does not include
a private address space.
This environment is private and unique to a task.
Each task has its own environment.
This task environment consists of a number of resources
(as represented in the TCB). Of interest in this discussion are the following.
Note that any of these task resources may be disabled in the NuttX
configuration to reduce the NuttX memory footprint:
1. **Environment Variables**. This is the collection of variable assignments
of the form: ``VARIABLE=VALUE``.
2. **File Descriptors**. A file descriptor is a task specific number
that represents an open resource (a file or a device driver, for example).
3. **Sockets**. A socket descriptor is like a file descriptor, but
the open resource in this case is a network socket.
4. **Streams**. Streams represent standard C buffered I/O.
Streams wrap file descriptors or sockets to provide a new set of interface
functions for dealing with the standard C I/O (like ``fprintf()``,
``fwrite()``, etc.).
In NuttX, a task is created using the interface ``task_create()``.
NuttX Task Exit Sequence
------------------------
.. figure:: task_exit_sequence.png
:alt: Task Exit Sequence diagram.
Task Exit Sequence diagram.
The Pseudo File System and Device Drivers
=========================================
A full discussion of the NuttX file system belongs elsewhere,
see :ref:`nuttx-filesystem` for more details.
But in order to talk about task resources, we also need to have
a little knowledge of the NuttX file system.
NuttX implements a Virtual Files System (VFS) that may be used to communicate
with a number of different entities via the standard ``open()``, ``close()``,
``read()``, ``write()``, etc, interfaces.
Like other VFSs, the NuttX VFS will support file system mount points,
files, directories, device drivers, etc.
Also, as with other VFSs, the NuttX file system will support
pseudo-file systems, that is, file systems that appear as normal media
but are really presented under programmatic control.
In Linux, for example, you have the ``/proc`` and the ``/sys``
psuedo-file systems.
There is no physical media underlying the pseudo-file system.
The NuttX root file system is always a psuedo-file system.
This is just the opposite from Linux. With Linux the root file system
must always be some physical block device (if only an initrd ram disk).
Then once you have mounted the physical root file system, you can mount
other file systems – including Linux pseudo-filesystems like ``/proc``
or ``/sys``.
With NuttX, the root file system is always
a pseudo-file system that does not require any underlying block driver
or physical device.
Then you can mount real filesystem in the pseudo-filesystem.
This arrangement makes life much easier for the tiny embedded world (but also
has a few limitations — like where you can mount file systems).
**NuttX interacts with devices via device drivers** – that is via software
that controls hardware and conforms to certain NuttX conventions
(see ``include/nuttx/fs/fs.h``). Device drivers are represented
by device nodes in the pseudo-file system.
By convention, these device nodes are created in the ``/dev`` directory.
Now that we have digressed a little to introduce the NuttX file system
and device nodes, we can return to our discussion of task resources.
``/dev/console`` and Standard Streams
-------------------------------------
There are three special cases of I/O: ``stdin``, ``stdout``, and ``stderr``.
These are type ``FILE*`` and correspond to file descriptors ``0``, ``1``,
and ``2`` respectively.
When the very first thread is created (called the IDLE thread),
the special device node ``/dev/console`` is opened. ``/dev/console`` provides
the ``stdin``, ``stdout``, and ``stderr`` for the initial task.
Inheritance of the Task Environment and I/O Redirection
=======================================================
When one task creates a new task, that new task inherits the task resources
of its parent. This includes all of the environment variables,
file descriptors, and sockets.
.. note::
Task resources inheritance can be limited by special options
in the NuttX configuration.
So, if nothing special is done, then every task will use ``/dev/console``
for the standard I/O. However, a task may close file descriptor
``0`` through ``2`` and open a new device for standard I/O.
Then any children tasks that are created will inherit that new re-directed
standard I/O as well.
This mechanism is used throughout NuttX.
For example in the THTTPD server to redirect socket I/O to standard I/O
for CGI tasks. In the Telnet server so that new tasks inherit the
Telnet session.
Tasks vs. Pthreads
==================
Systems like Linux also support POSIX pthreads.
In the Linux environment, the process is created with one thread running
in it. But by using interfaces like ``pthread_create()``, you can create
multiple threads that run and share the same process resources.
NuttX also supports POSIX pthreads and the NuttX pthreads also support
this behavior. That is, the NuttX POSIX pthreads also share the resources
of the parent task.
However, since NuttX does not support process address environments,
the difference is not so striking.
When a task creates a pthread, the newly create pthread will share
the environment variables, file descriptors, sockets, and streams
of the parent task.
.. note::
Task resources are reference counted and will persist
as long as a thread in the task group is still active.
See :ref:`tasks-vs-threads` for more details.
Process IDs / Task IDs / Pthread IDs
====================================
The term process ID is standard (usually abbreviated as pid) and used to
identify a task in NuttX. So, more technically, this number is a task ID
as was described above.
Pthreads are also described by a ``pthread_t`` ID.
In NuttX, the ``pthread_t`` ID is also the same task ID.

126
Documentation/implementation/oneshot_timers_and_cpu_load.rst Normal file
View file

@ -0,0 +1,126 @@
=======================================
Oneshot Timers and CPU Load Measurement
=======================================
Issues with CPU Load Measurement
================================
There is been support for CPU load measurement for a long time.
Enabled with:
.. code-block:: sh
CONFIG_SCHED_CPULOAD=y
CONFIG_SCHED_CPULOAD_TIMECONSTANT=2
The main problem with the existing logic is that it ran synchronously
with the system timer. That turns out to be useless for measurement of the
performance of some tasks.
Many tasks operate synchronously with the timer interrupt, i.e., timed event
is the stimulus for task execution.
Those tasks are always sampled at essentially the same point in the execution
profile leading to nonsense results.
In order to get believable results in all cases you would need to:
* Sample at times are completely random with respect to program behavior, AND
* Sample at a high rate or for a very long time.
External Clock
==============
In order to work around these things, an option to use an external clock
was added:
.. code-block:: sh
CONFIG_SCHED_CPULOAD_EXTCLK=y
CONFIG_SCHED_CPULOAD_TICKSPERSEC=77
This should work well if ``CONFIG_SCHED_CPULOAD_TICKSPERSEC`` is prime
and less than ``CONFIG_USEC_PER_TICK``.
However, you were completely on your own one how to implement this
external clock. But no longer.
Using a Oneshot Timer to Drive CPU Load Measurement
===================================================
Recently, a generic, platform-independent one-shot lower half interface
was developed. That interface is described in
``include/nuttx/timers/oneshot.h``. There are implementations of the one shot
timer lower half available for STM32, STM32L4, SAM4CM, SAMA5D3/4,
and SAMV71/SAME70.
And now there is also an OS component that can be initialized and used to
drive the CPU load sampling from a precision, high rate oneshot timer.
The implementation of that feature resides at
``sched/sched/sched_cpuload_oneshot.c`` and is enabled with:
.. code-block:: sh
CONFIG_ONESHOT=y
CONFIG_CPULOAD_ONESHOT=y
Perhaps with additional configuration options that are likely required
to enable board-specific oneshot timer lower-half support.
The oneshot timer must still be configured by board specific logic which
must call:
.. code-block:: c
void sched_oneshot_extclk(FAR struct oneshot_lowerhalf_s *lower);
To start the CPU load measurement.
``sched_oneshot_extclk()`` is prototyped in ``include/nuttx/clock.h``.
There is some example setup code in the NuttX simulation code at
``boards/sim/sim/sim/src/sim_bringup.c``.
Entropy
=======
Another recent addition to NuttX came from David Alessio.
David contributed support for ``/dev/urandom`` with a built-in XorShift128
pseduo-random number generator (PRNG).
We have detached the XorShift128 implementation from the ``/dev/urandom``
implementation and moved it to ``/libc/misc`` so that is it available
for other purposes.
In particular, there is an option to use the XorShift128 PRNG to add entropy
to the CPU load measurement of the oneshot timer.
That feature is enabled with:
.. code-block:: sh
CONFIG_CPULOAD_ONESHOT_ENTROPY=n
for ``n=1..30`` and disabled with ``CONFIG_CPULOAD_ONESHOT_ENTROPY=0``.
This value represents the number of bits of entropy that will be added
to the oneshot interval delays.
The oneshot timer will be set to the following interval each time
the oneshot timer is restarted is ``CONFIG_CPULOAD_ONESHOT_ENTROPY``:
.. code-block:: sh
CPULOAD_ONESHOT_NOMINAL \
- (CPULOAD_ONESHOT_ENTROPY / 2) \
+ error \
+ nrand(CPULOAD_ONESHOT_ENTROPY)
Where:
* ``CPULOAD_ONESHOT_NOMINAL`` is ``CONFIG_SCHED_CPULOAD_TICKSPERSEC``
in units of microseconds.
* ``CPULOAD_ONESHOT_ENTROPY`` is ``(1 << CONFIG_CPULOAD_ONESHOT_ENTROPY)``.
* ``error`` is an error value that is retained from interval to interval
so that although individual intervals are randomized,
the average will still be equal to ``CONFIG_SCHED_CPULOAD_TICKSPERSEC``.
If ``CONFIG_CPULOAD_ONESHOT_ENTROPY=0``, then the interval delay will
always be equal to ``CPULOAD_ONESHOT_NOMINAL``.

221
Documentation/implementation/power_management.rst Normal file
View file

@ -0,0 +1,221 @@
.. power-management:
================
Power Management
================
Power Management (PM) Subsystem
===============================
I would expect that the logic that performs clocking adjustments would utilize
the NuttX Power Management (PM) subsystem. That subsystem basically implements
a random walk. It collects and monitors system utilization information from
devices drivers. This utilization information comes from critical device
drivers via calls to::
void pm_activity(int domain, int priority);
This function is called by a device driver to indicate that it is performing
meaningful activities (non-idle). This increments an activity counter and will
prevent entering reduced power states.
For a system driver by a human interface, ``pm_activity()`` might be called
with touchscreen in puts and/or keypad inputs.
As long as the human is interacting with the device it remains non-idle.
When there is no further activity, the activity counter will decrease
and when it crosses a threshold, it will initiate a change to a lower power
usage state.
There are many actions that systems may take when the reduced power state
is entered. They may, for example, dim that backlight on a display or enable
MCU-specific reduced power consumptions modes. And for the purposes
of this discussion, they may also reduce the clocking to the CPU.
Alternative way to conserve power is using :ref:`tickless` with some
(dis)advantages you need to consider to know if it fits your needs.
.. _dynamic-clocking:
Dynamic Clocking
================
Most configurations are created with fixed clock configurations.
One strategy for power management is to use variable, dynamic clocking
where the CPU clock frequency is lowered during periods of inactivity.
Clock Update Function
---------------------
Variable clocking is not difficult to achieve but most ports are created
with fixed clock configuration using settings defined in the ``board.h``
header file. For example, Kinetis has::
void kinetis_clockconfig(void)
In order to make clocking variable a few things are needed:
* The initial board settings should still be used at power up,
but there needs to be a new clock initialization function,
perhaps ``kinetis_clock_update()`` that takes a structure containing
all of the clock settings.
* The existing ``kinetis_clockconfig()`` would need to be gutted,
most of the logic being moved to ``kinetis_clock_update()``.
``kinetis_clockconfig()`` would just create the structure using the constant
settings from the ``board.h`` header file then call
``kinetis_clock_update()`` to establish the initial clock configuration.
.. note:: Additional complexities may be associated with changing the clocking
other than just controlling dividers, PLLs, and clock sources. For example,
FLASH wait states. When increasing the clocking, the FLASH wait states must
be increased BEFORE reconfiguring the faster clocking.
But when reducing the clocking, FLASH wait states cannot be reduced until
AFTER reconfiguring the slower clocking.
Handling Driver Dependencies
----------------------------
The first part of the problem was modifying the clock configuration logic
to support variable configurations.
The second part of the problem is that now all of the devices that depend
on clocking such as the timer, the system timer, all MCU-specific timers,
UARTs, and perhaps other device need to be notified of the change
in clock frequency.
* The system timer, the SysTick in Cortex-M MCUs, needs to be notified
of the of the clock frequency change so that it can re-calculate
the system timer tick interrupt so that there is no disruption in timing.
* The same may be true of other MCU-specific timers. They may also need to
recalculate certain clocking. An MCU-specific timer may, for example,
be providing system time in Tickless mode.
* If serial devices are used, such as for a serial console, then
serial drivers must be notified of the clock change so that they can
recalculate their BAUD settings without lost of serial communications.
* Other devices such as SPI, I2C, I2S, SDIO, USB, etc. may also need to
be notified of the change in the clock configuration to behave correctly
across the clocking change.
Full functionality is not normally expected in low power consumption states.
So the most typical behavior for drivers in response to reduced power
consumption state changes is simply to shut themselves down (turn off
the peripheral, disable clocking to the peripheral, and other actions
as appropriate).
The above discussion only applies to peripherals that you expect
to continue normal operation in the reduced power state.
You may, as an example, want to keep timing accuracy throughout
the low power state or you may want to preserve serial debug output
in the reduced power state.
Driver PM Callback Functions
----------------------------
The NuttX PM subsystem provides the glue that integrates both parts
of the clock management problem.
The first part is managed by PM activity monitor and PM state changes.
The PM subsystem integrates the second part of the problem using PM driver
callback functions. The timing sensitive device drivers can be notified
of the change in clocking using the driver callbacks that are a port
of NuttX Power Management (PM) functionality.
Each timing sensitive device driver needs to register for a PM callback and,
on each callback, it must recalculate its frequency settings using the current
clock configuration.
To receive this callbacks, the driver logic needs to provide functions
with prototypes like:
.. code-block:: c
/* Power management callback function prototypes */
#ifdef CONFIG_PM
static void xyz_pm_notify(struct pm_callback_s *cb, int domain,
enum pm_state_e pmstate);
static int xyz_pm_prepare(struct pm_callback_s *cb, int domain,
enum pm_state_e pmstate);
#endif
Which are used to notify a callback vtable like:
.. code-block:: c
/* Power management callback function vtable */
#ifdef CONFIG_PM
static struct pm_callback_s g_xyz_pmcb =
{
.notify = xyz_pm_notify,
.prepare = xyz_pm_prepare,
};
#endif
The ``prepare()`` callback method gives the driver some advance notification
of the pending PM state change.
Some drivers may need to put the device in a safe state in the ``prepare()``
callback.
A serial driver, for example, may want to disable RX and TX so that garbage
is not generated or received during the clock change.
The ``notify()`` callback method signifies the actually state change
and it is in this method that the driver logic should recalculate its timing
parameters based on the new clock settings and also resume an operations
that were disabled by the ``prepare()`` callback.
The driver registers its PM callback vtable by calling pm_register() like:
.. code-block:: c
/* Register to receive power management callbacks */
int ret = pm_register(&g_xyz_pmcb);
Dynamic Clock Frequency Utility Functions
-----------------------------------------
Most timers and drivers in the system us the constant frequency values defined
in the ``board.h`` header file. In order to have variable time, the drivers
must be able to determine the current clock frequencies dynamically,
not via a fixed, constant definitions.
This means that there must be clock functions that derive various clock
firequencies in the clock distribution knowing only the input frequency
(i.e., crystal frequency of internal clock selection).
When the driver receives the PM ``notify()`` informing it that the clocking
may have changed, it must call these functions to get the new clock settings
instead of using the constant settings in board.h.
There is something similar in the source tree now for the case where NuttX
is started by a bootloader. In that case, the clock configuration is
set up by the bootloader, usually u-boot, and each driver that needs to know
clock frequencies must query the clock configuration to determine
the frequency.
For the SAMA5D4-EK, that is still handled by definitions in the ``board.h``
header file. See, for example, ``boards/arm/sama5/sama5d4-ek/include/*.h``.
``board.h`` includes another file that defines the clock setup based
on the configuration. If the SAMA5D4-EK is running out of SDRAM,
then it must have been started by a bootloader and the definitions appear in
``boards/ar/sama5/sama5d4-ek/include/board_sdram.h``.
In that case, the macros are redefined so that they map to a function call
to determine the clock frequency:
.. code-block:: c
#define BOARD_MAINCK_FREQUENCY BOARD_MAINOSC_FREQUENCY
#define BOARD_PLLA_FREQUENCY (sam_pllack_frequency(BOARD_MAINOSC_FREQUENCY))
#define BOARD_PLLADIV2_FREQUENCY (sam_plladiv2_frequency(BOARD_MAINOSC_FREQUENCY))
#define BOARD_PCK_FREQUENCY (sam_pck_frequency(BOARD_MAINOSC_FREQUENCY))
#define BOARD_MCK_FREQUENCY (sam_mck_frequency(BOARD_MAINOSC_FREQUENCY))
Something like this could also be done for other architectures to support
dynamic clock management.
You would probably have to do something similar if change the clocking.
``SysTick`` and each device would need a callback from the PM subsystem and each
would have to re-calculate is BAUD based on the new clock settings using
such functions calls to get the dynamic clock frequency.

BIN
Documentation/implementation/short_delay.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 22 KiB

300
Documentation/implementation/short_time_delays.rst Normal file
View file

@ -0,0 +1,300 @@
.. _short-time-delays:
=================
Short Time Delays
=================
System Timer Interrupt
======================
This section addresses some counter-intuitive properties of using very short
time delays.
This section assumes that timing is generated by a system timer
"tick" interrupt. In Tickless Mode there is no system timer interrupt.
Much of the discussion here would also apply in the Tickless mode, however,
the terminology used here assumes system timer interrupts.
Timer Resolution
================
When we talk about short delays, we are talking about delays that are
on the same order of magnitude as the system timer interrupt interval.
That interval is controlled the configuration setting
``CONFIG_USEC_PER_TICK``. The default value of ``CONFIG_USEC_PER_TICK``
is 10,000 microseconds.
That equivalent to a timer interrupt frequency of 100Hz:
.. code-block:: c
Ftimer ticks/sec = 1,000,000 usec/sec / CONFIG_USEC_PER_TICK;
There are many different OS interfaces that implement timed delays
(or function that have timeout values such as ``sem_timedwait()``).
These all behave in basically the same way.
.. _usleep:
``usleep()``
============
For simplicity of discussion let's focus on on ``usleep()``.
``usleep()`` is a standard but deprecated interface that simply delays
for a specified number of microseconds.
.. note:: ``usleep()`` is deprecated in favor of ``clock_nanosleep()``.
Requirements/Assumptions of ``usleep()``
----------------------------------------
The prototype for ``usleep()`` is:
.. code-block:: c
int usleep(useconds_t usec);
Where ``usec`` is the number microseconds to delay.
``usleep()`` will return zero unless an error occurs in which case
it will set the errno variable and return ``-1``.
Now the interesting questions:
Suppose that ``CONFIG_USEC_PER_TICK`` is set to 10,000 microseconds,
What will happen when you try any of the following?
.. code-block:: c
usleep(0);
usleep(1);
usleep(10000);
In order to predict such behavior, we will have to enumerate some of the
assumption and requirements of ``usleep()``:
1. The contract that ``usleep()`` makes is that it will suspend the calling
thread for at least usec microseconds. It will never, under any condition,
return with a delay smaller than the requestion usec delay.
2. ``usleep()`` may wait longer than the requested delay due to small system
processing overhead, task priorities, and quantization errors.
Task Priorities
---------------
When the requested delay expires, the calling task is ready to run but still
may not actually be able to run for some time due to higher priority tasks
that block execution of the ready to run task.
Quantization Errors
-------------------
``usleep()`` is only capable of waiting for multiples of the system timer
interrupt interval (``CONFIG_USEC_PER_TICK``) If the requested ``usec`` delay
is not an even multiple of the system timer interrupt interval,
then it will be rounded up as necessary to satisfy the first requirement above.
This means that the following are really equivalent delays:
.. code-block:: c
usleep(1);
usleep(10000);
And finally, the behavior that confuses most people..
.. note::
``usleep()`` has no knowledge of the the phase of the system timer
when it is started: The next timer interrupt may occur immediately
or may be delayed for almost a full cycle.
In order to meet the contract of the first requirement, the requested
time is also always incremented by one.
This means, for example, that the the delay:
.. code-block:: c
usleep(10000);
will not delay for one clock tick! That would be impossible!
There is no event exactly one clock tick after ``usleep()`` is called.
The next timer tick will always occur at some time strictly less than
the system timer interval.
Rather, ``usleep()`` will delay for two clock ticks resulting some
actual delay between 10 and 20 microseconds, exclusive.
See the following figure:
.. figure:: short_delay.png
:alt: Short Time Delays in NuttX.
Short Time Delays in NuttX.
For the most part, when delays are large – hundreds or thousands
of microseconds – this error is not significant.
It becomes noticeable only if you are using ``usleep()``
for delays very close to the the timer resolution when the error
can be relatively significant.
Finally, the easy one:
.. code-block:: c
usleep(0);
This will return immediately with no delay.
That satisfies all requirements and assumptions of the interface.
Using ``usleep()`` to implement periodic delays
-----------------------------------------------
The third assumption is necessary because ``usleep()`` has
no knowledge of the current system timer phase.
But it also makes ``usleep()`` a very bad choice for implementing
periodic behavior if the ``usleep()`` delay is close to the system
timer resolution.
For example, consider a loop such as the following
(again assuming that ``CONFIG_USEC_PER_TICK`` is set
to 10,000 microseconds):
.. code-block:: c
for (; ; )
{
usleep(10000);
/* Do some periodic stuff */
}
From the preceding discussion, we know that this will not work:
It will not (and cannot) create periodic processing at the rate
of the system timer interrupt rate but, rather, at half the system
time interrupt rate in this case.
In this case, that the third assumption of ``usleep()`` is not valid.
``usleep()`` does not start timing some random, unknown phase with respect
to the system timer interrupt.
In this case, ``usleep()`` will be started at a nearly constant phase
with respect to the system timer (at some some short delay after the
system timer interrupt) and will consistently show
near-worst case timing errors.
Of course, ``usleep()`` cannot know that and, hence, is a bad choice
for implementing such periodic behavior.
The user in this case is assuming that ``usleep()`` simply waits for the
next timer tick to occur. That is not the behavior or ``usleep()`` that
describes some non-existent interface that waits for the next timer interrupt
event, not for a fixed time delay.
.. note::
``usleep()`` behavior is to wait to assure that at least usec
microseconds has elapsed. And it does that job quite well.
A final note: The above periodic delay loop can be made to work well,
on the other hand, if the delay provided to ``usleep()`` is significantly
larger that the system timer resolution.
What can you do?
----------------
What can you do to improve the resolution for such high frequency processing
loop?
First, you might consider increasing the system timer interrupt rate.
You would do this by reducing the value of ``CONFIG_USEC_PER_TICK``.
For example a value of ``1000`` for ``CONFIG_USEC_PER_TICK`` would create
a timer interrupt rate of 1KHz and a minimum loop delay of
perhaps 2 milliseconds.
The trade-off here is that when you increase the timer interrupt rate,
the timer interrupt processing will then take a proportionately larger amount
of your CPU bandwidth.
The recommended way to get very high timer resolution without increasing
the timer interrupt rate (in most use cases) is to use a Tickless Mode OS.
For example, in the Tickless configuration, you could have
``CONFIG_USEC_PER_TICK`` set to ``1`` for a 1MHz timer resolution
(with no interrupts).
If your target periodic processing time is still 1Khz,
then this periodic processing could be met with good precision.
Another option is to abandon the system timer altogether and use
a dedicated timer peripheral to perform your timed operations with
high precision.
But if you really want to use the system timer than another thing
you should consider would be to implement a Timer Hook.
.. timer_hook:
Timer Hook
==========
A Timer Hook is a user provided function that is called from the OS
on each timer interrupt.
If you enable ``CONFIG_SYSTEMTICK_HOOK=y`` in your configuration,
then the OS timer interrupt handler will call out to a user-provided function,
``board_timerhook()``, on each timer interrupt.
The full prototype of this function is provided in ``included/nuttx/board.h``
as:
.. code-block:: c
void board_timerhook(void);
.. note::
The timer hook is only available when system timer interrupts
are used; it is not available in Tickless mode.
This timer hook could be used in a scenario where you would like to have
your task run at the each timer interrupt without the strange rounding
performed by the standard delay functions.
You might do something like the following as an example:
In your servicing task, implement a loop. At the top of the loop,
you would wait on a semaphore.
In the body of the loop you perform the periodic operation
then return to the wait at the top of the loop. Like:
.. code-block:: c
int ret;
while ((ret = sem_wait(&g_waitsem) >= 0)
{
/* Do periodic operations */
}
.. note::
If the return value from ``sem_wait()`` is negative then
some unusual event occurred. In normal cases the errno might either:
``EINTR`` meaning simply that the wait was awakened with a signal.
You can continue to wait in this case. Or ``ECANCELED`` meaning
that the thread has been cancelled and you should abort the periodic
operations.
In your ``boards/<arch>/<chip>/<board>/src/`` directory you would implement
``void board_timerhook(void)``.
The implementation of this function could consist of only a single line
of code: It could just post the semaphore on each timer interrupt,
waking of the loop in the servicing task. Like:
.. code-block:: c
void board_timerhook(void)
{
(void)sem_post(&g_waitsem);
}
This is very efficient. There is no context switch overhead at all
getting from the the timing interrupt to to the servicing task.
None at all other that the normal interrupt return logic.
The servicing task should be the highest priority task in the system
to assure that it is the one that runs immediately
when the timer interrupt returns.

192
Documentation/implementation/signal_handlers.rst Normal file
View file

@ -0,0 +1,192 @@
.. _signal-handlers:
===============
Signal Handlers
===============
Signals are used to exchange information between sending and receiving thread.
Sending Thread
==============
Posting Signals
---------------
These are the actions that run on the thread that posts the signal.
Signals are initiated in these ways:
* ``signal/sig_kill.c: nxsig_kill()``. The standard function ``kill()`` is a
simple wrapper around ``nxsig_kill()``. ``nxsig_kill()`` can be used
to send a signal to any task group. It simply sets up the call
to ``nxsig_dispatch()``.
* ``signal/sig_queue.c: nxsig_queue()``. The standard function ``sigqueue()``
is a simple wrapper around ``nxsig_queue()``. ``nxsig_kill()`` can be used
to send a signal to any task group, passing more information than
is possible with ``kill()``.
It again simply sets up the call to ``nxsig_dispatch()``.
* ``signal/sig_notification.c: nxsig_notification()``. This logic can also
generate signals via a call to ``nxsig_dispatch()``.
But this is part of the internal, NuttX signal notification system.
It sends signals to tasks via the work queue.
signal/sig_dispatch.c: ``nxsig_dispatch()``
-------------------------------------------
.. code-block:: c
int nxsig_dispatch(pid_t pid, FAR siginfo_t *info);
This is the front-end for ``nxsig_tcbdispatch()`` that should be typically
be used to dispatch a signal. If ``HAVE_GROUP_MEMBERS`` is defined,
then this function will follow the group signal delivery algorithms:
This front-end does the following things before calling
``nxsig_tcbdispatch()``:
1. With ``HAVE_GROUP_MEMBERS`` defined:
1. Get the TCB associated with the ``pid``.
2. If the TCB was found, get the group from the TCB.
3. If the PID has already exited, lookup the group that that was
started by this task.
4. Use the group to pick the TCB to receive the signal.
5. Call ``nxsig_tcbdispatch()`` with the TCB.
2. With ``HAVE_GROUP_MEMBERS`` not defined:
1. Get the TCB associated with the ``pid``.
2. Call ``nxsig_tcbdispatch()`` with the TCB
group/group_signal.c: ``group_signal()``
----------------------------------------
.. code-block:: c
int group_signal(FAR struct task_group_s *group, FAR siginfo_t *siginfo);
Send a signal to the appropriate member(s) of the group.
This is typically called from ``nxsig_dispatch()`` as described above
but may also be called from ``task/task_exithook.c`` to handle
Death-of-Child (``SIGCHLD``) signals.
group/group_signal.c: ``group_signal_handler()``
------------------------------------------------
.. code-block:: c
static int group_signal_handler(pid_t pid, FAR void *arg)
Callback from ``group_foreachchild()`` that handles one member of the group.
signal/sig_dispatch.c: ``nxsig_tcbdispatch()``
----------------------------------------------
.. code-block:: c
int nxsig_tcbdispatch(FAR struct tcb_s *stcb, siginfo_t *info)
All signals received the task (whatever the source) go through this function
to be processed. This function is responsible for:
1. Determining if the signal is blocked.
2. Queuing and dispatching signal actions
3. Unblocking tasks that are waiting for signals
4. Queuing pending signals.
This function will deliver the signal to the specific task associated
with the specified TCB.
This function is also called when the OS needs to deliver a signal
to a specific task.
It is normally only called via ``group_signal_handler()`` so that is follows
the rules of signal deliver in multi-threaded tasks.
But it is also called from a few other places:
1. ``pthread/pthread_condtimedwait.c: pthread_condtimedout().``
Used to wake-up a specific task waiting on a condition.
2. ``task/task_exithook.c: task_sigchild()``
Used to send the Death-of-Child signal, ``SIGCHLD``, to the parent task.
For unmasked signals that have a signal handler attached,
``nxsig_tcbdispatch()`` will call the architecture-specific interface,
``up_schedule_sigaction()``.
arch/xxx/src/xxx/up_schedsigaction.c: ``up_schedule_sigaction()``
-----------------------------------------------------------------
.. code-block:: c
void up_schedule_sigaction(struct tcb_s *tcb, sig_deliver_t sigdeliver)
Where sigdeliver is always the OS function ``nxsig_deliver()``.
This function is called by the OS when one or more signal handling actions
have been queued for execution for a specific task.
The architecture specific-code must configure things so that the sigdeliver
callback (i.e., ``nxsig_deliver()``) is executed on the thread specified
by tcb as soon as possible. This amounts to saving the signal handler state
information in the TCB so that when the receiving task next executes,
it will be the signal handler that runs, not the normal, uninterrupted thread.
One of the fixups performed by ``up_schedule_sigaction()`` is to force
the address of the signal handler to point to the trampoline function
``up_sigdeliver()``.
There are other special cases when the signal is generated from interrupt
handler or when the a task signals itself for some reason.
Those variations are not addressed here.
Receiving Thread
================
signal/sig_deliver.c: ``nxsig_deliver()``
-----------------------------------------
.. code-block:: c
void nxsig_deliver(FAR struct tcb_s *stcb)
This function is called on the thread of execution of the signal receiving task
when that task next runs again. It processes all queued signals then returns.
The mechanism by which a signal is deliver depends on the build configuration;
in PROTECTED and KERNEL build modes, it must go through a trampoline,
``up_signal_dispatch()`` to handle user-space signal actions.
``up_signal_dispatch()`` will drop from kernel mode to user mode,
then call the signal handler.
Otherwise, the signal handler is called directly from ``nxsig_deliver()``.
When the signal handler returns, the action is over and there is only
clean up to be done.
arch/xxx/src/xxx/up_sigdeliver.c: ``up_sigdeliver()``
-----------------------------------------------------
.. code-block:: c
void up_sigdeliver(void)
This is the a signal handling trampoline.
Logic in ``up_schedule_sigaction()`` forced the signal action to be
delivered to ``up_sigdeliver()``.
``up_sigdeliver()`` will do such things as:
1. Set up the state to return to the normal, uninterrupted thread
when the signal handler exits.
2. Make sure that the signal handler runs with interrupts enabled.
3. Invoke the signal handler.
When the signal handler returns this function will:
1. Free up resources committed by ``up_schedule_sigaction()``.
2. Restore the interrupt state.
3. And perform a context switch to return to the normal, uninterrupted thread.
When the uninterrupted thread is next suspended, the return will go back to
``nxsig_deliver()`` which will continue delivering signals.
(Hmmm.. shouldn't any other queued signal actions be handled first
before returning to the normal, uninterrupted thread?)

872
Documentation/implementation/smp.rst Normal file
View file

@ -0,0 +1,872 @@
.. _smp:
===============================
SMP (Symmetric MultiProcessing)
===============================
Definition
==========
According to Wikipedia:
"Symmetric multiprocessing (SMP) involves a symmetric
multiprocessor system hardware and software architecture where two or more
identical processors connect to a single, shared main memory, have full access
to all I/O devices, and are controlled by a single operating system instance
that treats all processors equally, reserving none for special purposes.
Most multiprocessor systems today use an SMP architecture.
In the case of multi-core processors, the SMP architecture applies to
the cores, treating them as separate processors.
(..)
SMP systems are tightly coupled multiprocessor systems with a pool
of homogeneous processors running independently, each processor executing
different programs and working on different data and with capability
of sharing common resources (memory, I/O device, interrupt system and so on)
and connected using a system bus or a crossbar."
-- Source: https://en.wikipedia.org/wiki/Symmetric_multiprocessing.
Development Status
==================
SMP support is complete and stable in NuttX on several multi-core platforms.
Enabling SMP
============
SMP can be enabled on NuttX with the following configuration settings:
* ``CONFIG_SMP`` - Enables support for Symmetric Multi-Processing (SMP)
on a multi-CPU platform.
* ``CONFIG_SMP_NCPUS`` - This value identifies the number of CPUs support
by the processor that will be used for SMP.
* ``CONFIG_SMP_IDLETHREAD_STACKSIZE`` - Each CPU will have its own IDLE task.
System initialization occurs on CPU0 and uses
``CONFIG_IDLETHREAD_STACKSIZE``.
This setting provides the stack size for the IDLE task on CPUS 1
through ``(CONFIG_SMP_NCPUS-1)``.
This section provides the origin design specification for the implemention.
As a result, you may find that the test uses future and conditional tenses
when describing the implementation of SMP on NuttX.
This design has been maintained and now reflects the current "as-built"
state of SMP in NuttX.
Design Requirements
===================
The basic design requirements are pretty simple:
1. Need to be able to bring up NuttX running on multiple CPUs.
2. Need data structures to manage multiple active tasks.
3. Need to be able to schedule tasks on other CPUs.
4. Need to be able to modify tasks running on other CPUs.
5. Need to be able to manage critical sections on all CPUs.
6. Need spinlocks to block on all CPUs in all cases: semaphore,
signal, message queue, etc.
7. Need to understand how some non-standard NuttX operations things
like disabling pre-emption work.
Data Structures
===============
Task Lists
----------
At the core of the NuttX design are data structures called
**Task Control Blocks** or just **TCB**.
These data structures contain everything-you-need-to-know about
a thread or task.
These TCBs are retained in lists within the RTOS.
The state of a thread or task is then determined by which list
the TCB resides in.
The Read-To-Run Task List
-------------------------
On such TCB list is of particular importance in the implementation of SMP.
That is the so-called ready-to-run list, ``g_readytorun``.
That list contains the TCB of every task or thread that is not blocked
in any way and so is, well, ready to run.
The ``g_readytorun`` is a prioritized list. The lowest priority task
is in the list is the one at the end of the list and that must always
by the IDLE task.
That is the only task/thread that is permitted to have priority 0.
The highest priority, read-to-run task is always at the head of
``g_readytorun`` and must be the currently executing task.
All other tasks after this is eligle to run, but not currently running.
The Assigned Task List
----------------------
In order to support SMP, the function of the ``g_readytorun`` list
must change. This ``g_readytorun`` should still exist but it should
now contain only:
1. Only tasks/threads that are eligible to run, but not currently running, AND
2. Tasks/threads that have not been assigned to a CPU.
For SMP support there should be an array of assigned tasks like:
.. code-block:: c
volatile dq_queue_t g_assignedtasks[CONFIG_SMP_NCPUS];
Where ``CONFIG_SMP_NCPUS`` is the configured number of CPUs supported
by the processors. As its name suggests, on ``g_assignedtasks`` queue for
``CPU n`` would contain only tasks/threads that are assigned to CPU n.
Threads would be assigned a particular CPU by one of two mechanisms:
1. (Semi-)permanently through an RTOS interfaces such as
``pthread_attr_setaffinity()``, OR
2. Temporarily through new scheduling logic.
Tasks/threads that are assigned to a CPU via an interface like
``pthread_attr_setaffinity()`` would never go into the ``g_readytorun`` list,
but would only go into the ``g_assignedtasks[n]`` list for the CPU n to which
the thread has been assigned.
Hence, the ``g_readytorun`` list would hold only unassigned tasks/threads.
An indication within the TCB would indicated whether or not a task/thread
is assigned to a CPU and, if so, which CPU it is assigned to.
Scheduling logic would temporarily assign a task or thread to a CPU.
The assignment is only temporary because state data in the TCB would indicate
that the task is unassigned when, hence, it could be returned
to the ``g_readytorun`` list later.
The assigned tasks lists lists would be prioritized.
The highest priority task, and the one currently executing on CPU n would be
the one at the head of ``g_assignedtasks[n]``.
Tasks after the active task are ready-to-run and assigned to this CPU.
The tail of this assigned task list, the lowest priority task,
is always the CPU's IDLE task.
The CPU n scheduling logic would execute whenever the currently running task
is removed from the head of ``g_assignedtasks[n]``.
The algorithm might be something like:
.. code-block:: c
/* Is the assigned task list for the CPU empty? */
if (g_assignedtasks[cpu].head == NULL)
{
/* No.. Is the task at the head of the assigned list for the CPU lower
* in priority that the current (unassigned) task at the head of the
* ready-to-run list?
*/
FAR struct tcb_s *rtcb = (FAR struct tcb_s *)g_readytorun.head ;
FAR struct tcb_s *atcb = (FAR struct tcb_s *)g_assignedtasks[cpu].head;
if (atcb->sched_priority < rtcb->sched_priority)
{
/* Remove the TCB from the head of the g_readytorun list. */
/* Add that TCB to the g_assignedtasks[cpu] list (it will go at the
* head of the list).
*/
}
/* Now activate the task at the head of the g_assignedtasks[cpu] list on
* the CPU.
*/
}
The Current Task
----------------
There is a lot of logic in the RTOS now that obtains the TCB for the currently
excuting task by examining the head of the ``g_readytorun`` list.
You will see this assignment in many places, both in the core OS logic
in ``nuttx/sched`` but also in architecture-specific logic under
``nuttx/arch``:
.. code-block:: c
FAR struct tcb_s *rtcb = this_task();
Where ``this_task()`` is a macro defined in ``nuttx/sched/sched.h``
and expands as follows:
.. code-block:: c
#define current_task(cpu) ((FAR struct tcb_s *)g_readytorun.head)
#define this_cpu() (0)
#define this_task() (current_task(this_cpu))
Of course, that would not work with the proposed changes.
We would need to then get the TCB of the currently executing task/thread
for CPU n from the head of ``g_assignedtasks[n]``.
I would propose a replacing the above assignment with a macro like
``current_task()`` where that macro might expand to:
.. code-block:: c
#ifdef CONFIG_SMP
# define current_task(cpu) ((FAR struct tcb_s *)g_assignedtasks[cpu].head)
# define this_cpu() up_cpu_index()
#else
# define current_task(cpu) ((FAR struct tcb_s *)g_readytorun.head)
# define this_cpu() (0)
#endif
#define this_task() (current_task(this_cpu))
where ``up_cpu_index()`` is some new MCU specific interface that will
return an index associated with the currently active CPU.
.. note::
This is a two step operations: Step 1. Get the CPU number and
Step 2: Use the CPU number as an index into the
``g_assignedtasks[]`` array of lists. **This must be atomic!**
The schedule should be locked to assure that the task
is not suspended after fetching the CPU number then restarted
on a different CPU to access the ``g_assignedtasks[]`` array
of lists.
The IDLE Task
-------------
Without SMP, the ``g_readytorun`` list always ends with the TCB of IDLE task.
It is always guaranteed to be at the end of the list because the list
is prioritized and because the IDLE task has an impossibly low priority
that no other task/thread could have.
The IDLE task is necessary because it gives the CPU something to execute
when there is nothing else to be done.
But with SMP, there are multiple CPUs that need something to do when there
is nothing else to do. We are tentatively thinking that each CPU needs its own
IDLE thread whose TCB would reside at the end of each
``g_assignedtasks[cpu]`` list. But that does feel wasteful
I am not certain the mechanism as of this writing, but I assume that
the ``nx_start()`` initialization logic would need to create an IDLE task
for each CPU and assign each IDLE task to each CPU.
CPU Index
---------
In order to access arrays indexed by a CPU ID value, some method must be
generated to provide the CPU ID that the currently executing task
is running on. To provide this index value, an interface
``up_cpu_index()`` is proposed.
For ARM, the implementation of ``up_cpu_index()`` can be accomplished
by reading the CP15 Multiprocessor Affinity Register (MPDIR).
That register has a 2 bit field index provides exactly the index that
we need for the SMP implementation.
Looking at how Linux does this, Linux uses an interface called ``get_cpu()``
which is analogous to the proposed ``up_cpu_index()``.
``get_cpu()`` maps to ``smp_processor_id()`` and if debug options are
not enabled, this further maps to ``raw_smp_procesor_id()``.
For the case of ARM, this maps to ``(current_thread_info()->cpu)``
where ``current_thread_info()`` is a location at the far end of the
allocated stack:
``(current_stack_pointer & ~(THREAD_SIZE-1))`` and
``THREAD_SIZE`` is ``(PAGE_SIZE << THREAD_SIZE_ORDER)``.
So, to make that long story short, Linux solves the problem by putting
some magic information at the base of far end of each stack when
a context switch occurs (and when the CPU is also known).
That magic information can then just be recovered using the thread's
stack pointer at any time. This is part of the basic implementation
of Thread Local Storage (TLS) in Linux.
Something similar could be done with NuttX and would require:
1. Special aligned stack allocation,
2. Logic to write the CPU index into the stack when each thread
is [re-]started.
This would also place an upper limit on the size of the stack:
If we are going to find the far end of the stack by simply ANDing out
the lower bits, then size of that mask would also determine
the maximum size of the stack.
However, I believe that using the information from the MPIDR register
is a better general solution. Counter-arguments are:
1. There may be some architectures that do not have such a simple mechanism.
2. TLS has value in any event.
3. The stack-based TLS is in user-accessible memory and could be used
by applications in protected and kernel builds.
System Startup
==============
I assume that initially, only one CPU is active.
System initialization would then occur on that single thread.
At the completion of the initialization of the OS, just before beginning
normal multitasking, the additional CPUs would be started.
Each CPU would be provided the entry point to is IDLE task when started.
Perhaps the MCU interface would be something like:
.. code-block:: c
int up_cpu_start(int cpu, main_t idletask);
The OS initialization logic would call this function repeatedly
until each CPU is started.
Scheduler Interactions
======================
In the general case, the scheduler should have full control over the current
state of all tasks. It must make that that if there are N CPUs that the top N
highest priority tasks are running.
Srict priority scheduling is the requirement, but perhaps the scheduling logic
could do some load balancing to distribute work as evenly as possible
over the CPUs. When a new task or thread becomes ready to run,
the scheduler must include some heuristics for assigning that task to a CPU
to achieve some optimal performance.
There are complications to how one CPU controls the tasks already running
on another CPU. To determine a task should run, you would need to be able to:
* Keep the task data structures stable while they are being analyzed.
* Find the lowest priority running task which could be on any CPU.
* If that priority is lower than the priority task, then replace it with
the new task at the head of the ``g_assignedtasks[]`` list.
* If not, find the task with the next lowest priority and compare that one.
* Continue until until the new task is assigned to a CPU or until
it is determined that all of the currently running tasks are higher priority
than the new task. In that base, the new task should be added
to the ``g_readytorun`` list.
To support this behavior, I think that the following new MCU interfaces
will be needed:
.. code-block:: c
int up_cpu_pause(int cpu);
Which would stop execution on CPU0, saving the state of the currently running
task so that it may be resumed. And:
.. code-block:: c
int up_cpu_resume(int cpu);
Restart the CPU with the task at the head of the ``g_assignedtasks[]`` list.
.. note::
Please also note the the "Signal Handling" paragraph below.
The same issue exists for dispatching signals to threads actively
running on another CPU.
Interrupt Handling
==================
Per-CPU Interrupts
------------------
How will interrupts be taken? On one CPU or on multiple CPUs?
This may work different on different hardware platforms.
This design requires only that:
* If the processor supports interrupts on only one CPU, then interrupts
cannot be nested; further interrupts must be disabled while that interrupt
handler runs (see Nested Interrupts and High Priority,
Zero Latency Interrupts.).
* If the process supports device interrupts on multiple CPUs, the interrupt
handling on the CPUs is not concurrent: When interrupts are disabled
on one CPU, they are disabled on all CPUs (unless, of course, if interrupts
are needed for inter-CPU communication).
However, I do not know of any CPU architecture that supports disabling
interrupts on one CPU from another CPU.
Instead, critical sections will need to be supported via spinlocks
as described below.
If interrupts can be taken by multiple CPUs then any data structures used
for interrupt handling would also need to become and array indexed by the
CPU number. Most architectures current use a data structure defined like:
.. code-block:: c
volatile uint32_t *g_current_regs;
Which would have to become an array like:
.. code-block:: c
volatile uint32_t *g_current_regs[CONFIG_SMP_NCPUS];
System Calls
============
System Calls are normally implemented via software interrupts.
The System Call software interrupt should run on the same CPU as does
the logic that generated the System Call or, alternatively,
the design must have some way of obtaining the index of the CPU
that generated the System Call.
Critical Sections
=================
A critical section is a set of statements that must be able to execute
exclusively. Higher level applications will, of course, use OS application
interfaces such as ``sem_wait()`` and ``sem_post()`` to manage critical
sections. But within the OS, for example, in the low level implementation
of ``sem_wait()`` and ``sem_post()``, more primitive, non-standard methods
must be used to implement critical sections.
.. _spinlocks:
Spinlocks
=========
A spinlock is a lock which causes a thread trying to acquire it to simply wait
in a loop (spin) while repeatedly checking if the lock is available.
The thread remains active but is not performing a useful task.
The use of such a lock is a kind of busy waiting and is used commonly
in SMP implementations to manage access to resources by multiple CPUs.
Spinlock Implementation
-----------------------
In a NuttX implementation, the spinlock would probably involve only:
* A memory location with one value, say ``SP_LOCKED``, meaning that the lock
is taken and another value, ``SP_UNLOCKED``, meaning that the lock
is available.
* An integer type memory location that contains the number of the CPU
holding the lock.
* An integer type memory location hold the number of counts on the lock.
* And a loop performs a test-and-set operation: The memory location
is read by a thread and set to true in one atomic operation.
If the read value is false, then the thread holds the lock.
Otherwise, it must loop trying repeatedly until the thread gets the lock.
The meaning of the lock is that CPU holding the lock has exclusive access
to a resource that is shared by multiple CPUs.
So there is never any reason for two threads on the same CPU to spin:
If the CPU already holds the lock, additional threads need simply
only increment the lock count.
* If the test-and-set fails in the logic that is spinning, but if the lock
is held by the logic that that CPU is running on, then the spin logic
should simply increment the count of locks (which needs to be atomic only
for single processor).
Could this cause one CPU to hog too much resource time?
Perhaps, been calls to the test-and-set logic, the spinlock should call
``sched_yield()`` which would at least let other threads
of the same priority run.
* Releasing the lock should be matter of decrementing the lock count
and if the lock count would decrement to zero, setting the lock value to
``SP_UNLOCKED``. This will, of course, allow another thread spinning
on the lock in a different CPU to take the lock for that CPU.
The following new, internal OS interfaces are proposed:
.. code-block:: c
void spin_lock(FAR spinlock_t *lock);
void spin_unlock(FAR spinlock_t *lock);
Where the type ``spinlock_t`` is defined in MCU-specific header files.
These new spinlock interfaces would also use the MCU-specific interface:
.. code-block:: c
spinlock_t up_testset(FAR spinlock_t *lock);
.. note::
A thread may take the lock while running on one CPU, but then later
be assigned to a different CPU, and then release the lock while
running on that other CPU. Is there a problem in this?
Yes, probably. One solution might be lock the thread
to a CPU if it holds the lock?
There is also a risk is that the thread holding the lock will be pre-empted
by the OS scheduler while holding the lock. If this happens, other threads
on other CPUs will be left spinning (repeatedly trying to acquire the lock),
while the thread holding the lock is not making progress towards releasing it.
The result is an indefinite postponement until the thread holding the lock
can finish and release it.
Spinlock logic can be common. However, there must be a unique instance
of that common spinlock logic in each OS operation that requires mutually
exclusive access by a CPU.
Now, what will we do with these spinlocks? Is there really a need for them?
Yes, probably. We will need examine every place in the OS that uses disabling
of pre-emption or disabling of interrupts to prevent other tasks
(and interrupts) from executing.
All of those cases need to be reconsidered and, most likely, protected
with spinlocks.
Let's next examine all of the cases of how resources are managed in NuttX.
Spinlocks in Semaphores, Signals, and Message Queues
----------------------------------------------------
A critical section using ``irqsave()`` and ``irqrestore()`` is already used
in the implementation of these inter-process communications to enforce
a critical section.
One a single CPU system, disabling interrupts will prevent context switches
(by prevent the asynchronous events that could cause a context switch)
and also prevents conflicts with interrupt level processing.
I believe that simply replacing ``irqsave()`` and ``irqrestore()`` with
new proposed functions ``enter_critical_section()`` and
``leave_critical_section()``, as described below under Disabling Interrupts,
should be sufficient.
These proposed functions include a spinlock to assure that they do enforce
a critical section.
Spinlocks and Data Caches
-------------------------
If spinlocks are used in a system with a data cache, then there may be
a problem with cache coherency in some CPU architectures.
When one CPU modifies the spinlock, the changes may not be visible
to another CPU if it does not share the data cache.
That would cause failure in the spinlock logic.
Flushing the D-cache on writes and invalidating before a read
is not a good option. Spinlocks are normally 8-bits in size and cache
lines are typically 32-bytes so that would have side effects unless
the spinlocks were made to be the same size as one cache line.
The better option is to add compiler independent "ornamentation"
to the spinlock so that the spinlocks are all linked together
into a separate, non-cacheable memory regions.
Because of region alignment and minimum region mapping sizes
this could still be wasteful of memory.
This would work in systems that have both data cache and either an MPU
(such as Cortex-m7) or an MMU (such as Cortex-Ax).
Disabling Pre-emption
=====================
Pre-emption is disabled via the interface ``sched_lock()``.
``sched_lock()`` currently works by preventing context switches from the
currently executing tasks.
This prevents other tasks from running (without disabling interrupts)
and gives the currently executing task exclusive access to the (single)
CPU resources.
Thus, ``sched_lock()`` and its companion, ``sched_unlcok()``,
are used to implement some critical sections.
Currnetly, Pre-emption is disabled using a simple lockcount in the TCB.
When the scheduling is locked, the lockcount is incremented;
when the scheduler is unlocked, the lockcount is decremented.
If the lockcount for the task at the head of the ``g_readytorun``
list has a ``lockcount > 0``, then pre-emption is disabled.
No special protection is required since only the executing task
can modify its lockcount.
Certainly, disabling context switches on one CPU would still be possible
in an SMP model, but it may not be possible to give a task exclusive access
to the (multiple) CPU resources without stopping the other CPUs:
Even though pre-emption is disabled, other threads will still be executing
on the other CPUS.
The full dynamics of the behavior of the scheduler logic in this case
is not certain.
However, I think that this would be an acceptable behavior provided that:
* There is a global lock count ``g_cpu_lockset`` that includes a bit
for each CPU: If the bit is ``1``, then the corresponding CPU has
the scheduler locked; if ``0``, then the CPU does not have the scheduler
locked.
* Scheduling logic would set the bit associated with the cpu in
``g_cpu_lockset`` when the TCB at the head of the
``g_assignedtasks[cpu]`` list transitions has ``lockount > 0``.
This might happen when ``sched_lock()`` is called, or after
a context switch that changes the TCB at the head of the
``g_assignedtasks[cpu]`` list.
* Similarly, the cpu bit in the global ``g_cpu_lockset`` would be cleared
when the TCB at the head of the ``g_assignedtasks[cpu]`` list has
``lockount == 0``. This might happen when ``sched_unlock()`` is called,
or after a context switch that changes the TCB at the head of the
``g_assignedtasks[cpu]`` list.
* Modification of the global ``g_cpu_lockset`` must be protected
by a simplified spinlock, ``g_cpu_schedlock``. That spinlock would be
taken when ``sched_lock()`` is called, and released when ``sched_unlock()``
is called. This assures that the scheduler does enforce the critical
section. NOTE: Because of this spinlock, there should never be more
than one bit set in ``g_cpu_lockset`` attempts to set additional bits
should be cause the CPU to block on the spinlock. However, additional
bits could get set in ``g_cpu_lockset`` due to the context switches
on the various CPUs.
* Each the time the head of a ``g_assignedtasks[]`` list changes
and the scheduler modifies ``g_cpu_lockset``, it must also set
``g_cpu_schedlock`` depending on the new state of ``g_cpu_lockset``.
* Logic that currently uses the currently running tasks lockcount
should instead use the global ``g_cpu_schedlock``.
A value of ``SP_UNLOCKED`` would mean that no CPU has pre-emption disabled;
``SP_LOCKED`` would mean that at least one CPU has pre-emption disabled.
Disabling pre-emption is a non-standard feature but the general capability
is common to many RTOS. But since feature is non-standard and perhaps
not realizable in the SMP model, another option would be
to simply eliminate it.
Disabling Interrupts
====================
Closely related to disabling pre-emption is the practice of disabling
interrupts to get exclusive access to resources.
Disabling interrupts is not really so different from disabling
pre-emption in practice.
It effectively disables pre-emption by preventing any asynchronous
events that could cause a context switch and, of course, in addition
prevents interrupt level processing.
So disabling of interrupts is also used in places to implement critical
sections and, because the similarity in behavior to disabling only
pre-emption, suffers from the same issues in the SMP environment.
Currently, interrupts on the single CPU are enabled and disabled with:
.. code-block:: c
irqstate_t irqsave(void);
void irqrestore(irqstate_t flags);
Those functions disable interrupts on the single CPU.
In the SMP environment, they would need to disable interrupts in all CPUs.
.. note::
The legacy ``irqsave()`` and ``irqrestore()`` have been replaced
with new functions implemented in the OS,
``enter_critical_section()`` and ``leave_critical_section()``.
These might be implemented as follows (highly simplified):
.. code-block:: c
spinlock_t g_spu_irqlock = SP_UNLOCKED;
#ifdef CONFIG_SMP
irqstate_t enter_critical_section(void)
{
irqstate_t flags = irqsave();
spinlock(&g_cpu_irqlock);
g_cpu_irqset |= (1 << cpu);
return flags;
}
void leave_critical_section(irqstate_t flags)
{
g_cpu_irqset &= ~(1 << cpu);
spinunlock(&g_cpu_irqlock);
irqrestore(flags);
}
There is an unhandled complexities in the above simplified logic.
Consider this scenario:
1. The thread calls ``enter_critical_section()``, disabling interrupts
on all CPUs and taking the spinlock.
2. The thread then suspends, waiting for an event. This is actually
a very standard behavior to suspend with interrupts disabled:
The system handles this gracefully be simply re-enabling interrupts
(if they were enabled by the next task to run).
3. Later, the event occurs, the task is again made ready-to-run,
and the interrupts are again disabled.
But,
1. There must be additional logic to release the spinlock
when the task is suspended.
2. There must be additional logic to re-acquire the spinlock
when the task restarts.
3. Is there any way that the spinlock could already be locked when the task
restarts? No, I don't think this is possible. If interrupts are disabled
and the spinlock is locked, then there should be no context switches.
There would be additional complexities if ``enter_critical_section()`` were
called during interrupt handling.
Interrupts are disabled during interrupt level processing, however, interrupt
level logic will attempt to establish critical sections even when
it does not need to do this: It will call ``enter_critical_section()`` anyway
because it will use some common logic with non interrupt level code.
There are many situations in which use of spinlocks as shown
in the simplified example will result in deadlock conditions.
As a result of these complexities, the full implementation of
``enter_critical_section()`` and ``leave_critical_section()`` are considerably
more complex.
See the logic in the file ``sched/irq/irq_csection.c`` if you are
really interested in the details.
Pre-Emption Controls and Critical Sections
==========================================
The effect of disabling pre-emption is to prevent to tasks from running
while on task has disabled pre-emption; the effect of entering a critical
section, on the other hand, is to:
1. Enforce exclusive access to the logic when in the critical section.
2. Keep the system stable while certain operations are performed.
3. Disable competing interrupt level activity when possible.
In order to keep the system stable within in the critical section
it is necessary, the critical section will modify the behavior of the
pre-emption controls.
The basic result is this modification is that new tasks are not permitted
to be started or resumed if:
1. Pre-emption is disabled, OR
2. Some other CPU other than the current CPU is in a critical section.
The CPU that has entered the critical section must have the ability
to start and stop tasks. Attempts to start new tasks from other CPUs when
one CPU is within the critical section is will result in the newly started
task being postponed in a pending task list, ``g_pendingtasks``.
Such pending tasks will only be allowed to run when:
1. All CPUs have re-enabled pre-emption, AND
2. All CPUs have left the critical section.
.. note::
It can be determined which CPU(s) have the critical section
by examining ``g_cpu_irqset``.
Signal Handlers
===============
There will be some issues related to how signals are delivered,
at least in regard to how signal handlers are executed.
I am thinking of the case where a signal is sent by a thread running
on one CPU to a thread running on another CPU that has a signal handler
installed. This would probably have to work as follows:
1. Stop the CPU on which the task is running ``using up_cpu_pause()``,
2. Schedule the signal action as is done in the existing logic, then
3. Re-start the CPU with ``up_cpu_resume()`` to resume execution with
the signal handler.
A special wrapper function for ``up_cpu_pause()`` is provided in the OS
to support this operation:
.. code-block:: c
int sched_tcb_pause(FAR struct tcb_s *tcb);
This function checks if the task associated with tcb is running on another CPU
and, if so, conditionally calls ``up_cpu_pause()`` to pause execution
on that CPU. It returns the CPU index of the paused CPU (or a negated
``errno`` value if no CPU was paused). While the CPU is paused, operations
can be performed on the data structures associated with the task.
Then the non-negative CPU index can then be used with
``up_cpu_resume()`` to restart the paused CPU.
This same sequence would have to be followed for other functions
that might need to modify the behavior of a running task such as
``task_delete()`` or ``task_restart()``.
Thread Affinity
===============
By default, a thread may run on any CPU.
There are some semi-standard interfaces that can be used to restrict
the set of CPUs that a thread may run on.
This set of CPUs is referred to as the threads affinity mask.
Semi-standard meaning used on Linux and available in ``GLIBC`` when
``__GNU_SOURCE`` is defined.
There are interfaces to set and get the affinity mask for a task prototyped
in ``sched.h``.
``cpusetsize`` is fixed in NuttX and must be equal to ``sizeof(cpu_set_t)``:
.. code-block:: c
#ifdef CONFIG_SMP
int sched_setaffinity(pid_t pid, size_t cpusetsize,
FAR const cpu_set_t *mask);
int sched_getaffinity(pid_t pid, size_t cpusetsize, FAR cpu_set_t *mask);
#endif
There are similar interfaces for a ``pthread`` prototyped in ``phtread.h``:
.. code-block:: c
#ifdef CONFIG_SMP
int pthread_setaffinity_np(pthread_t thread, size_t cpusetsize,
FAR const cpu_set_t *cpuset);
int pthread_getaffinity_np(pthread_t thread, size_t cpusetsize,
FAR cpu_set_t *cpuset);
#endif
.. note::
The ``_np`` in the naming is to remind you that
this **interface is non-POSIX**!
By default, a child task or pthread inherits the affinity mask of its parent.
The thread affinity mask for a ``pthread``, however, can also be set before
the thread is started via ``pthread_create()``:
.. code-block:: c
#ifdef CONFIG_SMP
int pthread_attr_setaffinity_np(FAR pthread_attr_t *attr,
size_t cpusetsize,
FAR const cpu_set_t *cpuset);
int pthread_attr_getaffinity_np(FAR const pthread_attr_t *attr,
size_t cpusetsize, cpu_set_t *cpuset);
#endif
In addition, macros are defined in the header file ``include/sched.h``
to abstract operations are CPU sets.
There are several such macros with names like ``CPU_ZERO()``, ``CPU_SET()``,
``CPU_CLR()``, etc.

794
Documentation/implementation/syslog.rst Normal file
View file

@ -0,0 +1,794 @@
.. _syslog:
======
SysLog
======
Standard SysLog Interfaces
==========================
The NuttX SYSLOG is an architecture for getting debug and status information
from the system. The syslogging interfaces are defined in the header file
``include/syslog.h``.
The primary interface to SYSLOG sub-system is the function ``syslog()`` and,
to a lesser extent, its companion ``vsyslog()``:
.. code-block:: c
/****************************************************************************
* Name: syslog and vsyslog
*
* Description:
* syslog() generates a log message. The priority argument is formed by
* ORing the facility and the level values (see include/syslog.h). The
* remaining arguments are a format, as in printf and any arguments to the
* format.
*
* The NuttX implementation does not support any special formatting
* characters beyond those supported by printf.
*
* The function vsyslog() performs the same task as syslog() with the
* difference that it takes a set of arguments which have been obtained
* using the stdarg variable argument list macros.
*
****************************************************************************/
int syslog(int priority, FAR const IPTR char *format, ...);
int vsyslog(int priority, FAR const IPTR char *src, va_list ap);
The additional ``setlogmask()`` interface can use use to filter
SYSLOG output:
.. code-block:: c
/****************************************************************************
* Name: setlogmask
*
* Description:
* The setlogmask() function sets the logmask and returns the previous
* mask. If the mask argument is 0, the current logmask is not modified.
*
* The SYSLOG priorities are: LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
* LOG_WARNING, LOG_NOTICE, LOG_INFO, and LOG_DEBUG. The bit corresponding
* to a priority p is LOG_MASK(p); LOG_UPTO(p) provides the mask of all
* priorities in the above list up to and including p.
*
* Per OpenGroup.org "If the maskpri argument is 0, the current log mask
* is not modified." In this implementation, the value zero is permitted
* in order to disable all syslog levels.
*
* REVISIT: Per POSIX the syslog mask should be a per-process value but in
* NuttX, the scope of the mask is dependent on the nature of the build:
*
* Flat Build: There is one, global SYSLOG mask that controls all output.
* Protected Build: There are two SYSLOG masks. One within the kernel
* that controls only kernel output. And one in user-space that controls
* only user SYSLOG output.
* Kernel Build: The kernel build is compliant with the POSIX requirement:
* There will be one mask for for each user process, controlling the
* SYSLOG output only form that process. There will be a separate mask
* accessable only in the kernel code to control kernel SYSLOG output.
*
****************************************************************************/
int setlogmask(int mask);
These are all standard interfaces as defined at https://www.OpenGroup.org.
Debug Interfaces
================
**In NuttX syslog output is really synonymous to debug output** and,
therefore, the debugging interface macros defined in the header file
``include/debug.h`` are also syslogging interfaces.
Those macros are simply wrappers around ``syslog()``.
.. note:: Debug here means "system log" rather than on-chip-debug.
The debugging interfaces differ from the syslog interfaces in that:
* They do not take a priority parameter; the priority is inherent
in the debug macro name.
* They decorate the output stream with information such as the file name.
* They can each be disabled via configuration options.
Each debug macro has a base name that represents the priority and a prefix
that represents the sub-system.
Each macro is individually initialized by both priority and sub-system.
For example, ``uerr()`` is the macro used for error level messages
from the USB subsystem and is enabled with ``CONFIG_DEBUG_USB_ERROR``.
The base debug macro names, their priority, and configuration variable
are summarized below:
* ``info()``: The ``info()`` macro is the lowest priority (``LOG_INFO``)
and is intended to provide general information about the flow of program
execution so that you can get an overview of the behavior of the program.
``info()`` is often very chatty and voluminous and usually more information
than you may want to see. The ``info()`` macro is controlled via
``CONFIG_DEBUG+subsystem+INFO``.
* ``warn()``: The ``warn()`` macro has medium priority (``LOG_WARN``)
and is controlled by ``CONFIG_DEBUG+subsystem+WARN``. The ``warn()`` is
intended to note exceptional or unexpected conditions that might be
potential errors or, perhaps, minor errors that easily recovered.
* ``err()``: This is a high priority debug macro (``LOG_ERROR``)
and controlled by ``CONFIG_DEBUG+subsystem+ERROR``.
The ``err()`` is reserved to indicate important error conditions.
* ``alert()``: The highest priority debug macro (``LOG_EMERG``)
and is controlled by ``CONFIG_DEBUG_ALERT``. The ``alert()`` macro
is reserved for use solely by assertion and crash handling logic.
It also differs from the other macros in that it is global
and cannot be enabled or disabled per subsystem.
SYSLOG Channels
===============
SYSLOG Channel Interfaces
-------------------------
In the NuttX SYSLOG implementation, the underlying device logic the supports
the SYSLOG output is referred to as a SYSLOG channel.
Each SYSLOG channel is represented by an interface defined
in ``include/nuttx/syslog/syslog.h``:
.. code-block:: c
/* This structure provides the interface to a SYSLOG device */
typedef CODE int (*syslog_putc_t)(int ch);
typedef CODE int (*syslog_flush_t)(void);
struct syslog_channel_s
{
/* I/O redirection methods */
syslog_putc_t sc_putc; /* Normal buffered output */
syslog_putc_t sc_force; /* Low-level output for interrupt handlers */
syslog_flush_t sc_flush; /* Flush buffered output (on crash) */
/* Implementation specific logic may follow */
};
The channel interface is instantiated by calling ``syslog_channel()``:
.. code-block:: c
/****************************************************************************
* Name: syslog_channel
*
* Description:
* Configure the SYSLOGging function to use the provided channel to
* generate SYSLOG output.
*
* Input Parameters:
* channel - Provides the interface to the channel to be used.
*
* Returned Value:
* Zero (OK)is returned on success. A negated errno value is returned
* on any failure.
*
****************************************************************************/
int syslog_channel(FAR const struct syslog_channel_s *channel);
``syslog_channel()`` is a non-standard, internal OS interface
and is not available to applications.
It may be called numerous times as necessary to change channel interfaces.
By default, all system log output goes to console (``/dev/console``).
SYSLOG Channel Initialization
-----------------------------
The initial, default SYSLOG channel is established with statically initialized
global variables so that some level of SYSLOG output may be available
immediately upon reset.
This initialized data is in the file ``drivers/syslog/syslog_channel.c``.
The initial SYSLOG capability is determined by the selected SYSLOG channel:
* In-Memory Buffer (**RAMLOG**). Full SYSLOG capability as available at reset.
* **Serial Console**: If the serial implementation provides the low-level
character output function ``up_putc()``, then that low level serial output
is available as soon as the serial device has been configured.
* For all other SYSLOG channels, all SYSLOG output goes to the bit-bucket
(discarded) until the SYSLOG channel device has been initialized.
The syslog channel device is initialized when the bring-up logic
calls ``syslog_intialize()``:
.. code-block:: c
/****************************************************************************
* Name: syslog_initialize
*
* Description:
* One power up, the SYSLOG facility is non-existent or limited to very
* low-level output. This function is called later in the initialization
* sequence after full driver support has been initialized. It installs
* the configured SYSLOG drivers and enables full SYSLOGing capability.
*
* This function performs these basic operations:
*
* - Initialize the SYSLOG device
* - Call syslog_channel() to begin using that device.
*
* If CONFIG_ARCH_SYSLOG is selected, then the architecture-specific
* logic will provide its own SYSLOG device initialize which must include
* as a minimum a call to syslog_channel() to use the device.
*
* Input Parameters:
* phase - One of {SYSLOG_INIT_EARLY, SYSLOG_INIT_LATE}
*
* Returned Value:
* Zero (OK) is returned on success; a negated errno value is returned on
* any failure.
*
****************************************************************************/
#ifndef CONFIG_ARCH_SYSLOG
int syslog_initialize(enum syslog_init_e phase);
#else
# define syslog_initialize(phase)
#endif
Different types of SYSLOG devices have different OS initialization
requirements. Some are available immediately at reset, some are available
after some basic OS initialization, and some only after OS is fully
initialized.
In order to satisfy these different initialization requirements,
``syslog_initialize()`` is called twice from the boot-up logic:
1. ``syslog_initialize()`` is called from the architecture-specific
``up_initialize()`` function as some as basic hardware resources
have been initialized: Timers, interrupts, etc.
In this case, ``syslog_initialize()`` is called with the argument
``SYSLOG_INIT_EARLY``.
2. ``syslog_initialize()`` is called again from ``nx_start()`` when
the full OS initialization has completed, just before the application
main entry point is spawned. In this case, ``syslog_initialize()``
is called with the argument ``SYSLOG_INIT_LATE``.
There are other types of SYSLOG channel devices that may require even further
initialization. For example, the file SYSLOG channel (described below)
cannot be initialized until the necessary file systems have been mounted.
Interrupt Level SYSLOG Output
-----------------------------
As a general statement, SYSLOG output only supports normal output from NuttX
tasks. However, for debugging purposes, it is also useful to get SYSLOG
output from interrupt level logic.
In an embedded system, that is often where the most critical operations
are performed.
There are three conditions under which SYSLOG output generated from interrupt
level processing can a included the SYSLOG output stream:
1. Low-Level Serial Output.
2. In-Memory Buffering.
3. Serialization Buffer.
The SYSLOG interrupt buffer is enabled with ``CONFIG_SYSLOG_INTBUFFER``.
When the interrupt buffer is enabled, you must also provide the size
of the interrupt buffer with ``CONFIG_SYSLOG_INTBUFSIZE``.
Low-Level Serial Output
^^^^^^^^^^^^^^^^^^^^^^^
If you are using a SYSLOG console channel (``CONFIG_SYSLOG_CONSOLE``)
with a serial console (``CONFIG_SYSLOG_SERIAL_CONSOLE``) and if the underlying
architecture supports the low-level ``up_putc()`` interface
(``CONFIG_ARCH_LOWPUTC``), then the SYLOG logic will direct the output
to ``up_putc()`` which is capable of generating the serial output
within the context of an interrupt handler.
There are a few issues in doing this however:
1. ``up_putc()`` is able to generate debug output in any context because
it disables serial interrupts and polls the hardware directly.
These polls may take many milliseconds and during that time, all interrupts
are disable within the interrupt handler. This, of course, interferes with
the real-time behavior of the RTOS.
2. The output generated by ``up_putc()`` is immediate and in real-time.
The normal SYSLOG output, on the other hand, is buffered in the serial
driver and may be delayed with respect to the immediate output by many
lines. Therefore, the interrupt level SYSLOG ouput provided throug
``up_putc()`` is grossly out of synchronization with other debug output.
In-Memory Buffering
^^^^^^^^^^^^^^^^^^^
If the RAMLOG SYSLOG channel is supported, then all SYSLOG output is buffered
in memory. Interrupt level SYSLOG output is no different than normal SYSLOG
output in this case.
Serialization Buffering
^^^^^^^^^^^^^^^^^^^^^^^
A final option is the use the an interrupt buffer to buffer the interrupt
level SYSLOG output. In this case:
1. SYSLOG output generated from interrupt level process in not sent
to the SYSLOG channel immediately. Rather, it is buffered in the
interrupt serialization buffer.
2. Later, when the next normal syslog output is generated, it will
first empty the content of the interrupt buffer to the SYSLOG device
in the proper context. It will then be followed by the normal syslog
output. In this case, the interrupt level SYSLOG output will interrupt
the normal output stream and the interrupt level SYSLOG output will
be inserted into the correct position in the SYSLOG output when
the next normal SYLOG output is generated.
SYSLOG Channel Options
======================
SYSLOG Console Device
---------------------
The typical SYSLOG device is the system console.
If you are using a serial console, for example, then the SYSLOG output
will appear on that serial port.
This SYSLOG channel is automatically selected by ``syslog_initialize()``
in the LATE initialization phase based on configuration options.
The configuration options that affect this channel selection include:
* ``CONFIG_DEV_CONSOLE``: This setting indicates that the system supports
a console device, i.e., that the character device ``/dev/console`` exists.
* ``CONFIG_SERIAL_CONSOLE``: This configuration option is automatically
selected when a UART or USART is configured as the system console.
There is no user selection.
* ``CONFIG_SYSLOG_CONSOLE``: This configuration option is manually selected
from the SYSLOG menu. This is the option that acutally enables the SYSLOG
console device. It depends on ``CONFIG_DEV_CONSOLE`` and it will
automatically select ``CONFIG_SYSLOG_SERIAL_CONSOLE`` if
``CONFIG_SERIAL_CONSOLE`` is selected.
* ``CONFIG_ARCH_LOWPUTC``: This is an indication from the architecture
configuration that the platform supports the ``up_putc()`` interface.
``up_putc()`` is a very low level UART interface that can even be used
from interrupt handling.
* ``CONFIG_SYSLOG_SERIAL_CONSOLE``: This enables certain features
of the SYSLOG operation that depend on a serial console.
If ``CONFIG_ARCH_LOWPUTC`` is also selected, for example,
then ``up_putc()`` will be used for the forced SYSLOG output.
Interrupt level SYSLOG output will be lost unless:
1. The interrupt buffer is enabled to support serialization, or
2. A serial console is used and ``up_putc()`` is supported.
.. note::
The console channel uses the fixed character device at
``/dev/console``. The console channel is not synonymous with
``stdout`` (or file descriptor ``1``). ``stdout`` is the
current output from a task when, say, ``printf()`` if used.
Initially, ``stdout`` does, indeed, use the ``/dev/console``
device. However, ``stdout`` may subsequently be redirected
to some other device or file.
This is always the case, for example, when a transient device
is used for a console – such as a USB console or a Telnet
console.
The SYSLOG channel is not redirected as ``stdout`` is; the SYSLOG
channel will stayed fixed (unless it is explicitly changed
via ``syslog_channel()``).
References: ``drivers/syslog/syslog_consolechannel.c`` and
``drivers/syslog/syslog_device.c``.
SYSLOG Character Device
-----------------------
The system console device, ``/dev/console``, is a character driver with
some special properties. However, any character driver may be used as the
SYSLOG output channel. For example, suppose you have a serial console
on ``/dev/ttyS0`` and you want SYSLOG output on ``/dev/ttyS1``.
Or suppose you support only a Telnet console but want to capture
debug output ``/dev/ttyS0``.
This SYSLOG device channel is selected with ``CONFIG_SYSLOG_CHAR`` and has
no other dependencies. Differences fromthe SYSLOG console channel include:
1. ``CONFIG_SYSLOG_DEVPATH``: This configuration option string must be set
provide the full path to the character device to be used.
2. The forced SYSLOG output always goes to the bit-bucket.
This means that interrupt level SYSLOG output will be lost unless
the interrupt buffer is enabled to support serialization.
3. ``CONFIG_SYSLOG_CHAR_CRLF``: If ``CONFIG_SYSLOG_CHAR_CRLF`` is selected,
then inefeeds in the SYSLOG output will be expanded to
Carriage Return + Linefeed. Since the character device is not a console
device, the addition of carriage returns to line feeds would
not be performed otherwise.
You would probably want this expansion if you use a serial terminal
program with the character device output.
References: ``drivers/syslog/syslog_devchannel.c`` and
``drivers/syslog/syslog_device.c``.
SYSLOG File Device
------------------
Files can also be used as the sink for SYSLOG output.
There is, however, a very fundamental difference in using a file as opposed
the system console, a RAM buffer, or character device:
You must first mount the file system that supports the SYSLOG file.
That difference means that the file SYSLOG channel cannot be supported during
the boot-up phase but can be instantiated later when board level logic
configures the application environment, including mounting of the file systems.
The interface ``syslog_file_channel()`` is used to configure
the SYSLOG file channel:
.. code-block:: c
/****************************************************************************
* Name: syslog_file_channel
*
* Description:
* Configure to use a file in a mounted file system at 'devpath' as the
* SYSLOG channel.
*
* This tiny function is simply a wrapper around syslog_dev_initialize()
* and syslog_channel(). It calls syslog_dev_initialize() to configure
* the character file at 'devpath then calls syslog_channel() to use that
* device as the SYSLOG output channel.
*
* File SYSLOG channels differ from other SYSLOG channels in that they
* cannot be established until after fully booting and mounting the target
* file system. This function would need to be called from board-specific
* bring-up logic AFTER mounting the file system containing 'devpath'.
*
* SYSLOG data generated prior to calling syslog_file_channel will, of
* course, not be included in the file.
*
* NOTE interrupt level SYSLOG output will be lost in this case unless
* the interrupt buffer is used.
*
* Input Parameters:
* devpath - The full path to the file to be used for SYSLOG output.
* This may be an existing file or not. If the file exists,
* syslog_file_channel() will append new SYSLOG data to the end of the
* file. If it does not, then syslog_file_channel() will create the
* file.
*
* Returned Value:
* Zero (OK) is returned on success; a negated errno value is returned on
* any failure.
*
****************************************************************************/
#ifdef CONFIG_SYSLOG_FILE
int syslog_file_channel(FAR const char *devpath);
#endif
References: ``drivers/syslog/syslog_filechannel.c``,
``drivers/syslog/syslog_device.c``, and ``include/nuttx/syslog/syslog.h``.
SYSLOG RAMLOG Device
--------------------
The RAMLOG is a standalone feature that can be used to buffer any character
data in memory. There are, however, special configurations that can be used
to configure the RAMLOG as a SYSLOG channel.
The RAMLOG functionality is described in a more general way
in the following paragraphs.
RAM Logging Device
^^^^^^^^^^^^^^^^^^
The RAM logging driver is a driver that was intended to support debugging
output (SYSLOG) when the normal serial output is not available.
For example, if you are using a Telnet or USB serial console,
the debug output will get lost – or worse.
For example, what if you want to debug the network over Telnet?
The RAM logging driver can also accept debug output data from interrupt
handler with no special serialization buffering.
As an added benefit, the RAM logging driver is much less invasive.
Since no actual I/O is performed with the debug output is generated,
the RAM logger tends to be much faster and will interfere much less
when used with time critical drivers.
The RAM logging driver is similar to a pipe in that it saves the debugging
output in a circular buffer in RAM.
It differs from a pipe in numerous details as needed to support logging.
This driver is built when CONFIG_RAMLOG is defined in the Nuttx configuration.
dmesg
^^^^^
When the RAMLOG (with SYSLOG) is enabled, a new NuttShell (NSH) command
will appear: ``dmesg``. The dmsg command will dump the contents of the
circular buffer to the console (and also clear the circular buffer).
RAMLOG Configuration options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* ``CONFIG_RAMLOG`` - Enables the RAM logging feature.
* ``CONFIG_RAMLOG_CONSOLE`` - Use the RAM logging device as a system console.
If this feature is enabled (along with ``CONFIG_DEV_CONSOLE``),
then all console output will be re-directed to a circular buffer in RAM.
This might be useful, for example, if the only console is a Telnet console.
Then in that case, console output from non-Telnet threads will go to the
circular buffer and can be viewed using the NSH ``dmesg`` command.
This optional is not useful in other scenarios.
* ``CONFIG_RAMLOG_SYSLOG`` - Use the RAM logging device for the syslogging
interface. If this feature is enabled, then all debug output will be
re-directed to the circular buffer in RAM. This RAM log can be viewed
from NSH using the ``dmesg`` command.
NOTE: Unlike the limited, generic character driver SYSLOG device,
the RAMLOG can be used to capture debug output from interrupt level handlers.
* ``CONFIG_RAMLOG_NPOLLWAITERS`` - The number of threads than can be waiting
for this driver on ``poll()``. Default: 4.
If ``CONFIG_RAMLOG_CONSOLE`` or ``CONFIG_RAMLOG_SYSLOG`` is selected,
then the following must also be provided:
* ``CONFIG_RAMLOG_BUFSIZE`` - The size of the circular buffer to use.
Default: 1024 bytes.
Other miscellaneous settings:
* ``CONFIG_RAMLOG_CRLF`` - Pre-pend a carriage return before every linefeed
that goes into the RAM log.
* ``CONFIG_RAMLOG_NONBLOCKING`` - Reading from the RAMLOG will never block
if the RAMLOG is empty. If the RAMLOG is empty, then zero is returned
(usually interpreted as end-of-file). If you do not define this, the NSH
``dmsg`` command will lock up when called! So you probably do want this!
* ``CONFIG_RAMLOG_NPOLLWAITERS`` - The maximum number of threads that
may be waiting on the poll method.
SYSLOG (Input) Character Device
===============================
If the option ``CONFIG_SYSLOG_CHARDEV`` is selected then support for a special
character device at ``/dev/syslog`` is supported.
The function ``syslog_register()`` can be used to register
that character device:
.. code-block:: c
/****************************************************************************
* Name: syslog_register
*
* Description:
* Register a simple character driver at /dev/syslog whose write() method
* will transfer data to the SYSLOG device. This can be useful if, for
* example, you want to redirect the output of a program to the SYSLOG.
*
* NOTE that unlike other syslog output, this data is unformatted raw
* byte output with no time-stamping or any other SYSLOG features
* supported.
*
****************************************************************************/
void syslog_register(void);
.. note:: Careful... there is overloaded naming here!
A character device can serve as the SYSLOG output channel as described above.
There we were referring to a data path like::
syslog() -> SYSLOG channel layer -> Character driver output channel.
Here we are talking about a different SYSLOG character devices that provides
input data to the SYSLOG channel.
That is a data path like::
SYSLOG Character driver -> SYSLOG channel layer -> Output channel.
Very confusing and begs for some re-naming.
Using SYSLOG for Debug
======================
Lately, I have starting thinking a little more about the ``SYSLOG`` functions
in general (as well as all of those debug macros). I think I am coming to the
conclusion that there needs to be some things done.
The original design of ``syslog()`` was just some hooks for simple serial
debug output. These simple debug hooks were standardized and renamed
``syslog()`` so that they provide a portable debug interface.
As NuttX has increased in complexity and sophistication over the years,
the implementation of ``syslog()`` "under the hood" is still that mindlessly
simple serial debug logic from the original NuttX release.
Perhaps the time as come to assess what is wrong with the solution
and to implement some improvements in the ``syslog()`` design?
Here are some of the issues of the current implementation that bother me:
Use of File Descriptors
-----------------------
In the default case, the debug output goes out on file descriptor ``1``
(``stdout``). But if you think about that, it is a little crazy.
Each task can have I/O redirected in various ways. In most simple systems,
a serial console is used for stdout in all tasks and the debug output goes
to the serial console so everything is seamless. But if you are using
a mixture of serial consoles, USB serial consoles, telnet sessions, etc.
then who know were where the output is going to go in the most general case.
Bits and pieces could go to different devices.
If you are redirecting stdout to a file, for example, then the debug
information could go into your file, corrupting the output that you wanted.
As another perverse example, try enabling network debug output using a Telnet
session. That is an interesting exercise for anyone who like to see
infinite loops: Telnet I/O generates debug output, the debug output goes
to the Telnet network connection, which generates more network debug output,
and on and one.
Interrupt Handlers
------------------
Output from interrupt handlers, of course, cannot use the console device
at all. File descriptor I/O is not permitted from interrupt handlers.
Attempts to do "normal" SYSLOG output from interrupt handlers will
just result in the output going to the bit bucket.
Low-Level Serial Driver
-----------------------
The usual workaround to get debug output from interrupt handlers is to use
the low-level serial I/O from interrupt handlers.
But there are issues with this as well:
* First, the console may not be the same serial device.
It might be something else althogether. That means that non-interrupt
SYSLOG output goes output one way and interrupt level SYSLOG output
always goes out the serial port.
Potentially very strange behavior could result.
* Second, the low-level serial output does a busy wait poll!
That interferes badly with the behavior of the interrupt handler –
it has to wait within the interrupt handler while the serial output
is performed. That wait would will be many milliseconds with
interrupts disabled!
* The low level serial output also interacts the serial driver itself
making other use of the console impossible. Even, in some cases
on some platforms, locking up the serial driver.
Interrupt Buffer
----------------
Another way to handle interrupt level output is supported.
This is the only other option available if a Serial Console is not being used.
This second option is enabled with ``CONFIG_SYSLOG_INTBUFFER``.
In this case, syslog output generated from interrupt level logic will simply
be buffered in memory. Then, later, when the next non-interrupt level syslog
output is generated, the buffer interrupt level output will performed.
This works because it essentially defers the syslog output generated
from interrupt handlers until the next opportunity to perform normal output.
Asynchronous Output
-------------------
Another syslog related issue is the asynchronous behavior with debug output
from interrupt handlers. The normal debug output goes to the serial driver
and is buffered for sending there. The size of the serial RX buffer
is configurable. At any given point in time, the current output is behind
realtme depending on that buffer size and the serial BAUD.
The interrupt debug output does not use the serial driver but immdiately
commandeers the serial port and outputs dara in real time.
The result is it appears to happen earlier in the output.
This is also why you lose the last debug output on a crash...
the last debug data is stranded in the serial drivers RX buffer.
Interleaved Output
------------------
Because the output is done character at a time, the debug output
from different tasks may get multiplexed and unreadable in the most critical
of cases. The interleaved output can become totally useless, usually
in the most complex situations where you need the debug output the most.
Many times my plans to debug a problem with SYSLOG output has been thwarted
because the output is just uninterpretable in a highly multitasking context.
Buffer Overrun
--------------
The root cause of this problem is the RX buffering in the serial driver:
Character output is done one character at a time. That is not usually
an issue. But when the system is very busy and he serial RX buffer becomes
full, then each character output causes the caller to suspending,
waiting for space in the RX buffer. It suspends and is moved to the last
of the FIFO for that priority.
When this happens, many tasks may be suspended waiting space to put the next
byte in the serial RX buffer. If the tasks are the same priority, then
the output will be interleaved as described since each task gets essentially
round-robin access to the serial driver. If one of the tasks is lower
priority, then its its output may be deferred for some time until all
of the higher priority tasks complete their output.
Again, leaving a big time skew in the output data.
When there is is a big time skew in the debug output – whether from
asynchronous output from interrupt handlers or from blocked,
lower priority tasks – This can lead to misinterpretation of the
debug output since our instinct is to treat the output as if it were
in sequential order in time.
.. note::
Although a simple working around for a partical debug scenario
is simply to increase the size of the RX buffer in the serial
driver. Assuming that the RX buffer overrun is only the result
of short-term, bursty behavior, then the large buffer migh prevent
tasks blocking waiting for space to write the next byte.
Of course, there is no work around for the perverse case where the debug
output is generated at a higher rate than can be transferred
on the serial port. You are just basically out-of-luck in that case.
Solutions
---------
Serialization Buffer
^^^^^^^^^^^^^^^^^^^^
Some of these asynchrony problems could also be reduced or eliminated
if all debug output were buffered in an in-memory FIFO. That buffer
would serialize output from diffrent sources:
Various tasks and interrupt level logic.
There is already a outgoing, RX buffer in the serial driver.
Why would an additional layer of buffering help? Only because then
the interrupt handler debug output could also be serialized.
The input to the FIFO is debug output from all concurrent tasks and interrupt
handlers; the single output of the FIFO would be the serial driver.
Serialization via syslog buffer has been recently implemented in NuttX,
this option is enabled with ``CONFIG_SYSLOG_BUFFER``.
Crash Dump
^^^^^^^^^^
It might even be possible to flush that serialization buffer at the time
of the crash. This has ability has not yet been implemented
as of the time of this writing.
CONFIG_SYSLOG and the RAMLOG
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There is a partial solution for all of these issues when ``CONFIG_SYSLOG=y``.
In that case, stdout is not used but instead some custom logic is used
that is established by the configuration. Currently that option is only used
with the ``RAMLOG``. The RAMLOG actually works very nicely and eliminates
most of the above issue (except for the interleaving issue).
But the ``RAMLOG`` also requires some additional logic to get the debug
information out of the ``RAMLOG``. In ``NSH``, you can use the dmesg command
to dump the content of the ``RAMLOG`` to the ``NSH`` console.
But I am thinking that some in-memory buffering and serialization
is the solution to most of the issues mentioned about:
That would end the reliance on stdout. It would be compatible with
debug output from interrupt handlers; interrupt handler output
would be serialized in the in-memory buffer in the correct position.
The same interleaving problem could still occur. Adding more buffering
can always eliminate the interleaving problem, provided that the problem
is due to bursty, high-volume output. But you can just increase the size
of the serial RX buffer to accomplish that; any special serialization
is of no help (other than it effectively increases the size of the RX buffer).
The existing ``RAMLOG`` code could be extended so that the buffered data
is agressively dumped to the ``SYSLOG`` device and I think all problems
would be eliminated. The buffer could be dumped that the end of each
non-interrupt level execution of ``syslog()`` (and also on a crash).
Interrupt level output would have to pend in the buffer until the next,
non-interrupt level debug output pushes it out to the serial driver.

BIN
Documentation/implementation/task_exit_sequence.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 82 KiB

217
Documentation/implementation/tasks_vs_threads.rst Normal file
View file

@ -0,0 +1,217 @@
.. _tasks-vs-threads:
=================
Tasks vs. Threads
=================
What is the difference between a thread and a task in Nuttx?
============================================================
Tasks and threads in NuttX try to emulate processes and threads
in the standard Unix environment.
I think of a process as a "container" of resources that are shared
by the threads that execute within the context of the process.
The process has one special thread, the "main" thread.
This is the special thread that is started when the process was created.
Since there are no processes in NuttX, the distinction is not so simple.
But in a similar way, tasks under NuttX have mostly private resources:
Their own file descriptors, their own ``FILE`` streams, their own ``errno``
variable, there own environment, etc.
Threads under NuttX, on the other hand, share task resources in the same way
that threads within the same process share the process resources.
The task resources were created when a task was created (``task_create()``,
``execv()``, ``posix_spawn()``, etc.).
When the task creates new ``pthreads``, those pthreads share the resources
of the parent task.
There is little to distinguish the main thread in NuttX.
All threads are essentially equal (as they were in old LinuxThreads library).
All task resources that are shared amongst threads reside in a "break-away",
reference-counted structure called struct ``task_group_s``.
The Task Control Block (TCB) of each thread that is a member of the task group
holds a reference to the same instance of this breakaway structure
(see ``include/nuttx/sched.h``).
The reference count of the shared,
task group resources is initialized to one when the task is created
(via ``task_create()``, for example); that task is the first member
of the task group.
The reference count is incremented when each additional thread is created
(via ``pthread_create()``) and joins task group.
The reference count equal to the number of members of the task group.
If any thread in the task group creates a new task (vs. thread),
that establishes a new task group but has no effect on the creator's
task group. The first task creates the task group;
the group membership only grows when threads are created via
``pthread_create()``.
As threads exit, they leave the task group and the reference count
on the struct ``task_group_s`` resources used by the thread is decremented.
When the reference count goes to zero, there are no members in the task group
and so the task group resources are finally deleted.
In this way, the life of a task resource begins when the task is created
and persists until the last thread in the task group exits.
Here is the definitive list of what is shared. Most of these resources
reside within the task group structure struct ``task_group_s``:
* Child task exit status.
* Pthread join data.
* Environment variables.
* File descriptors.
* FILE streams.
* Sockets.
* Opened message queues.
* ``pthread`` keys.
* Support data for ``atexit()``, ``on_exit()``, and/or ``waitpid()``.
The exception is the PIC address space used with NXFLAT.
It currently has its own data structure (struct ``dspace_s``,
but logically also belongs in ``task_group_s``):
* PIC data space and address environments.
.. note::
The ``errno`` is not one of the shared resources.
In NuttX ``errno`` is thread-private; each thread has
its own ``errno``. But for bug-for-bug compatibility, the ``errno``
variable really should also be moved into ``task_group_s``.
NuttX tasks and threads are discussed in much more detail in the
:ref:`nuttx-tasking` section.
When would I want to use a task? When would I want to use a thread?
===================================================================
Threads are very light weight; the memory cost for the thread is the cost
of the task control block plus the size of the stack.
That is perhaps 0.8-2KB, depending mostly on the configured stack size.
The memory cost of a task is higher because it includes most of the costs
of a pthread plus the cost of the task group structure.
The cost of the task group structure varies with such things as:
* The number of file descriptors you have configured (controlled by
``CONFIG_NFILE_DESCRIPTORS``),
* The number of streams that you have configured for C buffered I/O
(controlled by ``CONFIG_NFILE_STREAMS``),
* The size of the I/O buffer configured for for each stream
(controlled by ``CONFIG_STDIO_BUFFER_SIZE``). This setting is especially
important because there is an I/O buffer for each opened stream.
* The number of sockets that you have configured
(controlled by ``CONFIG_NSOCKET_DESCRIPTORS``).
* The size of dynamic data such as environment variables
(size is determined by your usage).
The size of task group can then become largely depending upon how these things
are configured, typically in the range 0.5-1KB.
So why would you ever use a task if pthreads use so much less memory?
**Pthreads and tasks share resources. So they are not independent.**
There is strong coupling between the threads in a task group
and what one thread does can affect the behavior of another thread.
Normally, each major block of functionality is implemented with a separate
task. With each of those ``_task group_s``, however, you may also want several
helper threads to implement asynchronous and concurrent behaviors.
How do signals work in a task group with many pthreads?
=======================================================
The behavior of signals in the multi-thread task group is complex.
NuttX emulates a process model with task ``group_s`` and follows the POSIX
rules for signalling behavior.
Normally when you signal the ``_task`` group you would signal using
the task ID of the main task that created the group (in practice, a different
task should not know the IDs of the internal threads created within
the task group); that ID is remembered by the task group
(even if the main task thread exits).
Here are some of the things that should happen when you signal
a multi-threaded task group:
* If a task group receives a signal then one and only one indeterminate thread
in the task group which is not blocking that signal will receive the signal.
* If a task group receives a signal and more than one thread is waiting
on that signal, then one and only one indeterminate thread out of that
waiting group will receive the signal.
You can mask out that signal using ``sigprocmask()``
(or ``pthread_sigmask()``). That signal will then be effectively disabled
and will never be received in those threads that have the signal masked.
On creation of a new thread, the new thread will inherit the signal mask
of the parent thread that created it.
So you if block signal signals on one thread then create new threads,
those signals will also be blocked in the new threads as well.
You can control which thread receives the signal by controlling
the signal mask.
You can, for example, create a single thread whose sole purpose
it is to catch a particular signal and respond to it:
Simply block the signal in the main task; then the signal will be blocked
in all of the pthreads in the group too.
In the one "signal processing" pthread, enable the blocked signal.
This thread will then be only thread that will receive the signal.
How do ``atexit()`` and ``on_exit()`` work with task groups?
============================================================
``atexit()`` and ``on_exit()`` callbacks must be registered using the task ID
of the main task (which makes sense since pthreads do not have task IDs
of type ``pid_t``).
The callback is not necessarily made when the task thread exits;
the ``atexit()`` and ``on_exit()`` callbacks will be executed when
the task group terminates, that is, when the final thread
of the task group terminates.
How does ``waitpid()`` work with task groups?
=============================================
In a single-thread task group, ``waitpid()`` will wait until the single,
main thread of the task group exits (i.e., the one created
by ``task_start()``). This is the intuitive behavior.
But the behavior may be less intuitive for multi-threaded task groups.
In a multi-threaded task group ``waitpid()`` will wait until
every thread in the task group exits.
Nothing special happens when the main thread of the task group exits.
What are privileged threads? How are threads handled differently when NuttX is built as a Kernel
================================================================================================
NuttX supports a build mode where it is built as a monolithic kernel.
This mode is selected with the configuration option
``CONFIG_BUILD_PROTECTED=y`` and is currently only supported
for a few architectures.
When used this way, NuttX is built as a separate kernel mode "blob"
and the applications are built as another separate user mode "blob".
The kernel runs in kernel mode and the applications run in user mode
(with the MPU restricting user mode accesses).
Access to the kernel from the user blob is only via system calls (SVCalls).
Thread and tasks that execute within the user mode "blob"
are all unprivileged, user mode threads. Exactly what "unprivileged"
means depends upon the memory protection architecture.
But it generally means that there are regions of memory where unprivileged
threads are prohibited from reading and/or writing data and/or from executing
code. Tasks created in the user-space are unprivileged;
all pthreads are unprivileged.
Certain threads are created within the kernel space to perform OS housekeeping
operations. Those are referred to as "kernel threads".
They are essentially the same as user-mode tasks but run in a privileged mode
and have full access to all of the restricted resources.

375
Documentation/implementation/tickless_os.rst Normal file
View file

@ -0,0 +1,375 @@
.. _tickless:
===========
Tickless OS
===========
Default System Timer
====================
By default, a NuttX configuration includes a periodic timer interrupt
that drives all system timing.
The timer is provided by architecture-specific code that calls into NuttX
at a rate controlled by ``CONFIG_USEC_PER_TICK``.
The default value of ``CONFIG_USEC_PER_TICK`` is 10000 microseconds which
corresponds to a timer interrupt rate of 100Hz.
On each timer interrupt, NuttX does these things:
* Increments a counter. This counter is the system time and has a resolution
of ``CONFIG_USEC_PER_TICK`` microseconds.
* Checks if it is time to perform time-slice operations on tasks that
have select round-robin scheduling.
* Checks for expiration of timed events.
What is wrong with this default system timer? Nothing really.
It is reliable and uses only a small fraction of the CPU band width.
But we can do better. Some limitations of default system timer are,
in increasing order of importance:
1. **Overhead:** Although the CPU usage of the system timer interrupt
at 100Hz is really very low, it is still mostly wasted processing time.
On most timer interrupts, there is really nothing that needs be done
other than incrementing the counter.
2. **Resolution:** Resolution of all system timing is also determined
by ``CONFIG_USEC_PER_TICK``. So nothing can be timed with resolution
finer than 10 milliseconds by default. If you want to increase this
resolution, then you can reduce ``CONFIG_USEC_PER_TICK``.
However, then the system timer interrupts use more of the CPU bandwidth
processing useless interrupts.
3. **Power Usage:** But the biggest issue is power usage.
When the system is IDLE, it enters a light, low-power mode (for ARMs,
this mode is entered with the ``wfi`` instruction for example).
But each interrupt awakens the system from this low power mode.
Therefore, higher rates of interrupts cause greater power consumption.
Tickless OS Advantages
======================
The so-called **Tickless OS** provides one solution to issue.
The basic concept here is that the periodic, timer interrupt is eliminated
and replaced with a one-shot, interval timer.
It becomes event driven instead of polled: The default system timer
is a polled design. On each interrupt, the NuttX logic checks if it needs
to do anything and, if so, it does it.
Using an interval timer, you can anticipate when the next interesting
OS event will occur, program the interval time and wait for it to fire.
When the interval time fires, then the scheduled activity is performed.
Platform Support
================
In order to use the Tickless OS, you must provide special support
from the platform-specific code. Just as with the default system timer,
the platform-specific code must provide the timer resources to support
the OS behavior.
Currently these timer resources are only provided on a few platforms.
You can see that implementation for the simulation in
``nuttx/arch/sim/src/up_tickless.c``. There is another implementation
for the Atmel SAMA5 at ``nuttx/arch/arm/src/sama5/sam_tickless.c``.
This Wiki will explain how you can provide the Tickless OS support
for your platform.
Configuration Options
=====================
* ``CONFIG_ARCH_HAVE_TICKLESS``: If your platform provides support
for the Tickless OS, then you should set this option in the ``Kconfig``
file for your board. Here is what the selection looks in the
``arch/Kconfig`` file for the simulated platform::
config ARCH_SIM
bool "Simulation"
select ARCH_HAVE_TICKLESS
---help---
Linux/Cywgin user-mode simulation.
* ``CONFIG_SCHED_TICKLESS``: If ``CONFIG_ARCH_HAVE_TICKLESS`` is selected,
then you will be able to use this option to enable the Tickless OS features
in NuttX.
* ``CONFIG_SCHED_TICKLESS_ALARM``: The tickless option can be supported either
via a simple interval timer (plus elapsed time) or via an alarm.
The interval timer allows programming events to occur after an interval.
With the alarm, you can set a time in the future and get an event when
that alarm goes off. This option selects the use of an alarm.
The advantage of an alarm is that it avoids some small timing errors;
the advantage of the use of the interval timer is that the hardware
requirement may be less.
* ``CONFIG_USEC_PER_TICK``: This is not a new option, but changes
its relevance when the Tickless OS is selected.
In the default configuration where system time is provided by a periodic timer
interrupt, the default system timer is configure the timer for 100Hz or
``CONFIG_USEC_PER_TICK=10000``. If ``CONFIG_SCHED_TICKLESS`` is selected,
then there are no system timer interrupt. In this case,
``CONFIG_USEC_PER_TICK`` does not control any timer rates.
Rather, it only determines the resolution of time reported by
``clock_systimer()`` and the resolution of times that can be set for certain
delays including watchdog timers and delayed work.
In this case there is still a trade-off:
It is better to have the ``CONFIG_USEC_PER_TICK`` as low as possible
for higher timing resolution. However, the the time is currently held
in ``unsigned int``. On some systems, this may be ``16-bit`` in width
but on most contemporary systems it will be ``32-bit``.
In either case, smaller values of ``CONFIG_USEC_PER_TICK`` will reduce
the range of values that delays that can be represented.
So the trade-off is between range and resolution (you could also modify
the code to use a 64-bit value if you really want both).
.. note:: Recently NuttX switched to ``int64_t`` for time stamps.
The default, 100 microseconds, will provide for a range of delays
up to 120 hours.
This value should never be less than the underlying resolution of the timer.
Error may ensue.
System Interfaces
=================
Implementation Notes
--------------------
Notice that with the ``CONFIG_SCHED_TICKLESS`` option, an implementation
might require two hardware timers:
(1) An interval timer to satisfy the requirements for ``up_timer_start()``
and ``up_timer_cancel()``, and
(2) a counter to handle the requirement of ``up_timer_gettime()``.
Since timers are a limited resource, the use of two timers could be an issue
on some systems. The job could be done with a single timer if, for example,
the single timer were kept in a free-running at all times.
Some timer/counters have the capability to generate a compare interrupt when
the timer matches a comparison value but also to continue counting without
stopping. If your hardware supports such counters, one might used the
``CONFIG_SCHED_TICKLESS_ALARM`` option and be able to simply set the
comparison count at the value of the free running timer PLUS the desired
delay. Then you could have both with a single timer:
An alarm and a free-running counter with the same timer!
Imported Intefaces
------------------
The interfaces that must be provided by the platform specified code
are defined in ``include/nuttx/arch.h`` and summarized below.
``up_timer_intialize()``
^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: c
void up_timer_initialize(void);
* **Description:** Initializes all platform-specific timer facilities.
This function is called early in the initialization sequence by
``up_intialize()``. On return, the current up-time should be available
from ``up_timer_gettime()`` and the interval timer is ready for use
(but not actively timing).
* **Input Parameters:** None.
* **Returned Value:** None.
* **Assumptions:** Called early in the initialization sequence before
any special concurrency protections are required.
``up_timer_gettime()``
^^^^^^^^^^^^^^^^^^^^^^
.. code:: c
int up_timer_gettime(FAR struct timespec *ts);
* **Description:** Return the elapsed time since power-up (or, more correctly,
since ``up_timer_initialize()`` was called). This function is functionally
equivalent to ``clock_gettime()`` for the clock ID ``CLOCK_MONOTONIC``.
This function provides the basis for reporting the current time and also
is used to eliminate error build-up from small errors
in interval time calculations.
* **Input Parameters:** ``*ts`` - Provides the location in which
to return the up-time.
* **Returned Value:** Zero (``OK``) is returned on success;
a negated ``errno`` value is returned on any failure.
* **Assumptions:** Called from the the normal tasking context. The implementation must provide whatever mutual exclusion is necessary for correct operation. This can include disabling interrupts in order to assure atomic register operations.
``up_alarm_cancel()``
^^^^^^^^^^^^^^^^^^^^^
.. code:: c
int up_alarm_cancel(FAR struct timespec *ts);
* **Description:** Cancel the alarm and return the time of cancellation
of the alarm. These two steps need to be as nearly atomic as possible.
``sched_timer_expiration()`` will not be called unless the alarm
is restarted with ``up_alarm_start()``. If, as a race condition,
the alarm has already expired when this function is called,
then time returned is the current time.
* **Input Parameters:** ``*ts`` - Location to return the expiration time.
The current time should be returned if the timer is not active.
``ts`` may be ``NULL`` in which case the time is not returned.
* **Returned Value:** Zero (``OK``) is returned on success;
a negated ``errno`` value is returned on any failure.
* **Assumptions:** May be called from interrupt level handling or from the
normal tasking level. Interrupts may need to be disabled internally
to assure non-reentrancy.
.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM``
is defined.
``up_alarm_start()``
^^^^^^^^^^^^^^^^^^^^
.. code:: c
int up_alarm_start(FAR const struct timespec *ts);
* **Description:** Start the alarm. ``sched_timer_expiration()`` will be
called at alarm expires (unless ``up_alarm_cancel()`` is called to stop
the alarm.
* **Input Parameters:** ``*ts`` - The time in the future at the alarm
is expected to occur. When the alarm occurs the timer logic will call
``ched_timer_expiration()`` is called.
* **Returned Value:** Zero (``OK``) is returned on success;
a negated ``errno`` value is returned on any failure.
* **Assumptions:** May be called from interrupt level handling
or from the normal tasking level. Interrupts may need to be disabled
internally to assure non-reentrancy.
.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM``
is defined.
``up_timer_cancel()``
^^^^^^^^^^^^^^^^^^^^^
.. code:: c
int up_timer_cancel(FAR struct timespec *ts);
* **Description:** Cancel the interval timer and return the time remaining
on the timer. These two steps need to be as nearly atomic as possible.
``sched_timer_expiration()`` will not be called unless the timer
is restarted with ``up_timer_start()``. If, as a race condition,
the timer has already expired when this function is called,
then that pending interrupt must be cleared so that
``up_timer_start()`` and the remaining time of zero should be returned.
* **Input Parameters:** ``*ts`` - Location to return the remaining time.
Zero should be returned if the timer is not active. ``ts`` may be ``NULL``
in which case the time remainging is not returned
* **Returned Value:** Zero (``OK``) is returned on success;
a negated ``errno`` value is returned on any failure.
* **Assumptions:** May be called from interrupt level handling
or from the normal tasking level. Interrupts may need to be disabled
internally to assure non-reentrancy.
.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM``
is not defined.
``up_timer_start()``
^^^^^^^^^^^^^^^^^^^^
.. code:: c
int up_timer_start(FAR const struct timespec *ts);
* **Description:** Start the interval timer. ``sched_timer_expiration()``
will be called at the completion of the timeout (unless
``up_timer_cancel()`` is called to stop the timing.
* **Input Parameters:** ``*ts`` - Provides the time interval until
``sched_timer_expiration()`` is called.
* **Returned Value:** Zero (``OK``) is returned on success;
a negated ``errno`` value is returned on any failure.
* **Assumptions:** May be called from interrupt level handling or from the
normal tasking level. Interrupts may need to be disabled internally
to assure non-reentrancy.
.. note:: This function is only required when ``CONFIG_SCHED_TICKLESS_ALARM``
is not defined.
Exported Interfaces
-------------------
In addition, the following interface is provided by the RTOS for use
by the platform specific code:
``sched_alarm_expiration()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: c
void sched_timer_expiration(FAR const timespec *tc);
* **Description:** if ``CONFIG_SCHED_TICKLESS`` is defined, then this
function is provided by the RTOS base code and called from
platform-specific code when the alaram used to implement
the tick-less OS expires.
* **Input Parameters:** ``*ts`` - The time when the alarm expired.
* **Returned Value:** None.
* **Assumptions:** Base code implementation assumes that this function
is called from interrupt handling logic with interrupts disabled.
.. note:: This function is only provided when ``CONFIG_SCHED_TICKLESS_ALARM``
is defined.
``sched_timer_expiration()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: c
void sched_timer_expiration(void);
* **Description:** if ``CONFIG_SCHED_TICKLESS`` is defined, then this
function is provided by the RTOS base code and called from
platform-specific code when the interval timer used to implement
the tick-less OS expires.
* **Input Parameters:** None.
* **Returned Value:** None.
* **Assumptions:** Base code implementation assumes that this function
is called from interrupt handling logic with interrupts disabled.
.. note:: This function is only provided when ``CONFIG_SCHED_TICKLESS_ALARM``
is not defined.
Issues
======
Let me describe an experience to illustrate some of the problems
that you may run into:
I implemented the tickless mode on an MCU with ``16-bit`` timers.
Most of the clock sources were too fast for use with the 16-bit timer,
so I tried using a 32.768 kHz clock source.
This resulted in a time resolution of about 30.518 microseconds.
That time, 30.518 microseconds, can not be represented accurately with
``CONFIG_USEC_PER_TICK`` so I used the value 31 which is in error
by about 0.6%.
What I found on a busy system is that the delay I get when executing
this NSH command was actually about 5.5 seconds::
nsh> sleep 10
This was very puzzling to me and cost me a lot of debug time.
Finally, I disabled all competing usage of the interval timer by disabling
features (the high priority work queue, by the way, is the most significant
user of the interval timer).
After disabling all competing usage, the NSH command sleep 10 resulted
in a delay of about 10.3 seconds. Still not accurate but not so bad.
I concluded that the gross inaccuracy in the first case was due to
the inaccuracies in the representation of the clock rate.
30.518 usec cannot be represented accurately.
Each timing calculation results in a small error.
When the interval timer is very busy, long delays will be divided
into many small pieces and each small piece has a large error
in the calculation. The cumulative error is the cause of the problem.
.. caution:: The moral of this story: Do not use timer sources
that cannot be represented by ``CONFIG_USEC_PER_TICK``.

358
Documentation/implementation/tls.rst Normal file
View file

@ -0,0 +1,358 @@
.. _thread-local-storage:
=========================
TLS: Thread Local Storage
=========================
Historical Background
=====================
Thread Local Storage (TLS) was originally implemented to support
pthread-specific data and a per-thread ``errno`` variable.
``errno``
=========
The user ``errno`` value was originally kept in the TCB and could be accessed
via an OS system call called ``__errno()``:
.. code-block:: c
# define errno *__errno()
The ``errno`` value was kept in the thread's TCB in the ``pterrno`` field.
The ``__errno()`` function finds the TCB associated with the current thread
and returns a pointer to that errno storage location.
This worked in the FLAT build mode because it dereferenced the address
of the errno in TCB as both an ``RVALUE``:
.. code-block:: c
errcode = errno;
And as an ``LVALUE``:
.. code-block:: c
errno = errcode;
That works, however, it is rather inefficient.
However, that will not work at all in the PROTECTED or KERNEL build modes
because the memory holding the TCB is protected and cannot be directly
accessed from the user mode application.
Instead, it is accessed through accessor functions:
.. code-block:: c
void set_errno(int errcode);
int get_errno(void);
Accessing the ``errno`` via these functions in PROTECTED or KERNEL mode
is a huge performance hit, because the calls must go through a system call
which is implemented as a software interrupt.
And in this case, the errno is defined to be:
.. code-block:: c
# define errno get_errno()
This works fine as an ``RVALUE``:
errcode = errno;
But will cause a compilation error if used as an ``LVALUE``:
.. code-block:: c
errno = errcode;
Instead user code must explicitly call ``set_errno(errcode)``.
This a a violation of the accepted usage of the POSIX ``errno`` variable.
TLS-Based ``errno``
-------------------
The current solution has moved the ``errno`` storage location out of the
protected TCB memory and into the unprotected application thread's stack
memory. Then it can be accessed with the appropriate TLS
(Thread Local Storage) interfaces.
Task-Specific Globals in the FLAT and PROTECTED builds
======================================================
A special case is the use of TLS to support task-specific
(vs. thread specific) global data.
Task-specific differs from thread-specific in that all of the threads
in the task group share the global data, however, each task group has
its own set of task-specific global variables.
This feature permits a high-fidelity emulation of process global data
as you would see in the KERNEL build where each process has its own copy
of all global variables.
Task-specific data is implemented by keeping such globals in the stack
of the main thread. A single main thread is always present and, hence,
provides user-accessible, per-task storage.
Task-specific data is accessed like TLS except that the task-specific data
is accessed from the stack of the main thread,
rather than the stack stack of the main thread.
Re-Entrant ``getopt()``
=======================
One use of task-specific data is for global variables used by the ``getopt()``
function. ``getopt()`` is an important C library function used by applications
for parsing of task command line parameters.
It is, however, not re-entrant due to the use of global variables:
``optarg``, ``opterr``, ``optind``, and ``optopt``.
There are several, non-standard implementations for a re-entrant version
of called ``getopt_r()``, however, these are all wildly different
and do not conform to any standard.
.. important:: Non-standard interfaces are not desirable in NuttX.
In the FLAT and PROTECTED builds, this non-reentrant limitation
poses a problem. Unlike the KERNEL build mode, there is one instance
of of the globals ``optarg``, ``opterr``, ``optind``, and ``optopt``
shared across all task groups.
It is not a problem in the KERNEL build where there is a separate instance
of the globals in each process address space.
Using task-specific data, however, it is possible to make ``getopt()``
thread-safe. This amounts to keeping the ``getopt()`` globals in the main
thread's TLS so that there is an individual copy for each thread.
That would keep both the standard form of the ``getopt()`` interfaces
as well as making ``getopt()`` full re-entrant with respect to task groups.
Unaligned TLS
=============
Historically, TLS worked by aligning the stack base address then
simply AND'ing the current stack pointer to obtain the base address
of the stack where the TLS data can be found (as struct ``tls_info_s``).
This mechanism is very efficient because no OS system call is required,
as application logic can obtain the TLS data directly form its stack
via AND'ing and casting.
This works well in the KERNEL build mode where the stack resides at highly
aligned virtual address, but does not work well in FLAT and PROTECTED modes:
1. The alignment must be large. That is because it also determines
the maximum size of the thread's stack. If the size of the thread's stack
exceeds the maximum value determined by the alignment, then
the AND operation will alias to the wrong address.
2. If all of the addresses are highly aligned then
(1) you need to have much more memory available for stack allocation.
This is because (2) the large alignment causes bad memory fragmentation
and degraded use of memory.
An alternative way to get the stack base address is to call into the OS
to get the unaligned stack base address.
This involves a system call, but is more usable that other alternatives
in the FLAT and PROTECTED modes.
We have implemented the configuration: ``CONFIG_TLS_ALIGNED`` that selects
the legacy aligned stack for TLS access. If this is not defined, then
the new unaligned stack TLS logic is used.
We have also implemented aligned and unaligned TLS support
for every architecture.
Addition implementation steps:
1. Moved the ``errno`` storage location out of the TCB and in TLS
(into the struct ``tls_info_s``).
2. Modified the ``errno`` access definitions and logic that was
in ``sched/errno`` to use the TLS logic in user space.
That logic now resides in ``libs/libc/errno`` since it is now
a user library interface, not a core OS interface.
3. TLS is now enabled by default. It is enabled in the unaligned mode
for the FLAT and PROTECTED build modes but in the highly efficient
aligned mode for KERNEL build mode.
4. There are no longer an OS system calls related to the error
(with the exception of a call to get the struct ``tls_info_s``
in the unaligned TLS mode).
Push-Up vs. Push-Down Stacks
============================
TLS data is always located at the beginning thread's stack.
This is true for both CPUs with push-up stacks and CPUs with push-down stacks.
This location required in order to access the TLS by ANDing the aligned stack
pointer address.
The stack memory maps, differ only in the usage of the available stack::
Push Down Push Up
+-------------+ +-------------+ <- Stack memory allocation
| TLS Data | | TLS Data |
+-------------+ +-------------+
| Task Data* | | Task Data* |
+-------------+ +-------------+
| Arguments | | Arguments |
+-------------+ +-------------+ |
| | | | v
| Available | | Available |
| Stack | | Stack |
| | | |
| | | |
| | ^ | |
+-------------+ | +-------------+
\*) Task data is allocated in the main's thread's stack only
TLS interfaces
==============
TLS is a non-standard, but more general interface.
It differs from pthread-specific data only in that its semantics are general;
the semantics of the pthread-specific data interfaces are focused on pthreads.
But they really should share the same common underlying logic.
Currently, there are four TLS interfaces:
.. code-block:: c
int tls_alloc(void);
int tls_free(int tlsindex);
uintptr_t tls_get_value(int tlsindex);
int tls_set_value(int tlsindex, uintptr_t tlsvalue);
as prototyped and documented in ``include/nuttx/tls.h``.
These interfaces are basically adaptations from the Windows TLS interfaces
which are directly analogous to the POSIX pthread-specific data interfaces,
adapted to NuttX coding standards (see, for example, links available at
https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-tlsalloc
).
There are no POSIX TLS interfaces and Linux (actual GLIBC) does not provide
a good mechanism. It relies on a storage class that is specific
to ELF binaries. That is not useful in an embedded system where
no ELF information is present.
pthread-specific data
=====================
Pthread-specific data is another mechanism for accessing thread-specific data.
It consists of these POSIX standard interfaces:
.. code-block:: c
int pthread_key_create(FAR pthread_key_t *key,
CODE void (*destructor)(FAR void *));
int pthread_setspecific(pthread_key_t key, FAR const void *value);
FAR void *pthread_getspecific(pthread_key_t key);
int pthread_key_delete(pthread_key_t key);
This, historically, was separate from TLS data and managed internal
to the OS. pthread-specific data was only accessible through OS system calls.
But it is a natural extension of the TLS logic to support both
non-standard TLS access and standard pthread-specific data access.
This would amount to:
1. Moving the pthread-specific storage out of the TCB
and into the TLS data.
2. Moving the pthread-specific data accessors out of ``sched/pthread``
and into ``libs/libc/pthread``.
3. Remove the pthread-specific data system calls.
Accessing the ``errno`` from kernel space
=========================================
.. important:: We should never access the errno from kernel space.
This is especially dangerous from kernel space because we can't be certain
which user address environment is in effect or which stack is being used
(if we were to use separate kernel mode stacks in the future as Linux does,
for security reasons, etc).
Most OS interfaces have two parts:
1. A user callable function, say ``osapi()`` that sets the ``errno`` value
is necessary (and may implement cancellation points) and
2. An internal OS version which might then be ``nx_osapi()`` which
does not modify the ``errno`` value.
Ideally, all of the user callable OS interfaces (the ``osapi()``) should be
moved to the location in ``libs/libc`` the system call (``sycall/``) should be
redirected to ``nx_osapi()``.
A complication to doing this is that the OS interfaces which are also
cancellation points call special internal interfaces,
``enter_cancellation_point()`` and ``leave_cancellation_point()``
that are not available from user space.
So some addition partitioning would be need to accomplish this.
This separation of user and OS interfaces has not been implemented
as of this writing.
Wild Ideas
==========
The primary beneficiary of TLS is the C library.
**The TLS interfaces are non-standard and should not be used directly
by any portable application code.**
However, these non-standard TLS interfaces can be used within the C library
to implement improved, standard user interfaces.
Local Process ID
----------------
Another data item that should, eventually, be included in the TLS data
is the process ID (``pid``) of the currently executing thread.
In some analysis of PROTECTED and KERNEL builds, it was found that
``getpid()`` was the most highly accessed OS interface.
By moving the PID into TLS, we could eliminate this system call overhead
(at least in the aligned TLS case).
The same mechanism would be used by ``pthread_self()`` which, in NuttX,
would be the equivalent function, but following pthread semantics.
Streams
-------
In NuttX, C buffered I/O streams are represented with a pre-group array
of type FILE. A stream is accessed from the OS via the
``nxched_get_streams()`` interface (system call).
The streams array is not used by OS and would be another candidate
to move into TLS.
.. note:: In retrospect, this is not such a good idea because the I/O streams
are not unique per-thread; they are common across all threads within
a task group: That includes main thread of the group and all child
pthreads whose parent, grandparent, of great-grandparent is the
main thread. They all share the same I/O stream array.
As a result, I/O streams must be one per task-group.
This works now because the I/O stream array is a part
of the internal OS group structure which is one per task group.
While it would be nice to get the I/O stream around out of the OS,
TLS is probably not the right solution to do that.
Code References
===============
Things to look at:
* ``include/errno.h`` - Defines current errno access.
* ``include/nuttx/tls.h`` - Defines the tls_info_s structure.
* ``include/nuttx/sched.h`` - Group related TLS structures.
* ``libs/libc/errno`` - The TLS-based errno logic.
* ``libs/libc/tls`` - The implementation of the most TLS interfaces.
* ``sched/group`` - Group-related implementation of certain TLS interfaces.
* ``libs/libc/pthread`` - The implementation of the pthread-specific dta interfaces.

147
Documentation/implementation/usb.rst Normal file
View file

@ -0,0 +1,147 @@
==========================
USB (Universal Serial Bus)
==========================
NAKing USB OUT/IN Tokens
------------------------
USB supports a NAKing mechanism to pace incoming OUT data from the USB host.
The USB device class driver can control this NAKing only indirectly
by the manner in which it manages read requests.
Similarly, USB will refuse to perform the USB host's requests for IN data
by NAKing the host's IN tokens.
.. note:: USB direction naming is host-centric. This makes life difficult for
people working USB device drivers:
IN refers to data coming from the device INto the host.
But for the device, this is a write operation (write being
peripheral-centric: from-memory-to-peripheral).
OUT refers to data sent from the host OUT to the device.
For the device this is a read (peripheral-to-memory) operation.
NAKing USB OUT Tokens
^^^^^^^^^^^^^^^^^^^^^
Below is how the NAKing is implemented in the interaction between the USB
device class driver and USB Device Controller Drivers (DCDs):
1. At initialization time when the USB device class driver is enumerated
by the host, the class driver will allocates a number of read requests
(which contain request buffers).
2. The USB device class driver sends each allocated read request to the device
DCD via ``DRVR_EPSUBMIT()``.
3. Upon receipt, the DCD will keep all of these empty read requests in a list.
4. When the OUT token is received what happens next depends upon the state
of the list of empty read requests:
4a. If there are no requests in the list of empty read requests, the DCD
will have configured the endpoint to NAK the OUT token. This NAKing
will continue until the list of read requests is non-empty and the DCD
disables the NAKing to signal that it can accept the OUT packet data.
4b. If the list of read requests is not empty, the DCD will remove the next
read request from the list, receive the DATA accompanying the OUT
token, copy the received packet data into the read request's buffer,
and return the filled reqd request to the USB device class driver.
The data is returned using a callback function pointer that is
contained inside of the read request.
5. When the filled read request if returned, the USB device class driver will
add the newly filled read request to the TAIL of a list of filled read
requests. This will then be available to the the driver read logic when
the application requests more data.
This step will also wake up read logic that has been suspended waiting
for the availability of incoming data.
6. The receive logic starts when the application requests data (or it is
resumed after the receipt of new data) it will look at the read request
at the HEAD of the list of filled read requests. If will return the data
payload to the application by copying it from the read request's buffer
into the application-provided receive buffer:
6a. If the application's receive buffer becomes full and it cannot accept
more data, the receive logic will save the offset in the data buffer
where it left off. If the application asks for more data, it will
resume transferring at the point where it left off.
6b. If the receive logic transfers all of the data from the receive
buffer. It will remove the read request from the HEAD of the list
of filled read requests and return the receive buffer to the driver
via ``DRVR_EPSUBMIT()`` (See step 3).
The receive logic will then continue at step 6.
6c. If the receive logic needs more data in order to complete
the application read operation, it will wait until it is awakened
by the receipt of a newly filled.
CDC/ACM driver
~~~~~~~~~~~~~~
The CDC/ACM driver differs from all other drivers because it is REQUIRED
to discard data if there is no application ready to receive the data.
This is how a serial device works. So there is nothing like step 5
in the CDC/ACM driver and in step 6 the CDC/ACM driver will not remember
the point where it left off; if the application cannot accept any further
data, it will return the partially emptied read request to the DCD
immediately to be filled – dropping an untransferred data into the bit bucket.
This is not an error or an oversight in the CDC/ACM. This is the intentional
design. This is the CORRECT behavior of the CDC/ACM driver. Data overrun can
only be prevented on a serial connection by using either XON/XOFF software
flow control or hardware flow control. A special endpoint 0 control request
is reserved for the CDC/ACM implementation.
.. note:: As of this writing, the logic that implements the CDC/ACM
_hardware flow control has not been implemented.
There is a place holder for such logic, but the implementation
is missing._
NAKing USB IN Tokens
^^^^^^^^^^^^^^^^^^^^
The DCD will NAK IN tokens whenever it has nothing to send to the host PC.
Without going into as much detail, suffice it to say that the logic flow
is very similar to the case off OUT tokens with the following differences:
1. The list of empty write requests are not provided to the DCD at
initialization time, but are instead are retained in a list of empty write
requests in the USB device class driver.
2. When the USB class driver has data to send, it will remove a write request
from this list, copy the outgoing application data into the write buffer,
and send the filled write request to the DCD via ``DRVR_EPSUBMIT()``.
If there are no available write requests, the application thread may block
waiting for the next write request to be returned from the DCD.
3. The DCD will keep a list of filled write requests for each endpoint.
When that list becomes non-empty, the DCD will disable the NAKing and
set up to transfer data to the host. Normally this involves copying
the IN data into an outgoing FIFO or setting up some write DMA transfers:
3a. After the data has been transferred to the USB device hardware,
the emptied write request will be returned to the USB device class driver.
This is done through a callback function pointer within the write request.
3b. If the DCD's list of write requests becomes empty, it will set up
the endpoint to NAK any further IN tokens (after the final IN transfer
completes).
4. When the USB device class driver receives the emptied write request from
the DCD, it will return the write request to its list of empty write
request and may, perhaps, wake up any thread that was waiting
for an available write request to send more data.
So the fundamental different between IN and OUT processing is that for
IN transfers, empty write requests are saved in the USB device class driver
and then given to the DCD when they are filled with outgoing data.
For OUT transfers empty read requests are retained in the DCD and returned
to the USB device class driver when they filled with incoming data.
Similarly, filled read requests are queued for application read processing
in the USB device class driver; filled write requests are queued for transfer
to the host in the DCD.
But in either case, NAKing occurs when associated DCD request queue is empty.

2
Documentation/reference/user/13_boardctl.rst
View file

@ -1,3 +1,5 @@
.. _board-ioctl:
===========
Board IOCTL
===========