diff --git a/Documentation/components/filesystem/how_nxffs_works.rst b/Documentation/components/filesystem/how_nxffs_works.rst new file mode 100644 index 00000000000..6afb0048228 --- /dev/null +++ b/Documentation/components/filesystem/how_nxffs_works.rst @@ -0,0 +1,11 @@ +=============== +How NXFFS works +=============== + +.. warning:: + Migrated from: + https://cwiki.apache.org/confluence/display/NUTTX/How+NXFFS+Works + +.. image:: nxffs/nxffs_page_0.png +.. image:: nxffs/nxffs_page_1.png +.. image:: nxffs/nxffs_page_2.png \ No newline at end of file diff --git a/Documentation/components/filesystem/index.rst b/Documentation/components/filesystem/index.rst index 798e85f8320..019787e35eb 100644 --- a/Documentation/components/filesystem/index.rst +++ b/Documentation/components/filesystem/index.rst @@ -553,6 +553,10 @@ NuttX provides support for a variety of file systems out of the box. userfs.rst zipfs.rst inotify.rst + nuttxfs.rst + nxflat.rst + pseudofs.rst + special_files_dev_num.rst FS Categories ------------- diff --git a/Documentation/components/filesystem/nuttxfs.rst b/Documentation/components/filesystem/nuttxfs.rst new file mode 100644 index 00000000000..c2fcb56bf74 --- /dev/null +++ b/Documentation/components/filesystem/nuttxfs.rst @@ -0,0 +1,11 @@ +======== +Nuttx FS +======== + +.. warning:: + Migrated from: + https://cwiki.apache.org/confluence/display/NUTTX/NuttX+File+System + +.. image:: nuttxfs/nuttxfs_page_0.png +.. image:: nuttxfs/nuttxfs_page_1.png +.. image:: nuttxfs/nuttxfs_page_2.png \ No newline at end of file diff --git a/Documentation/components/filesystem/nuttxfs/nuttxfs_page_0.png b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_0.png new file mode 100644 index 00000000000..2c22481f105 Binary files /dev/null and b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_0.png differ diff --git a/Documentation/components/filesystem/nuttxfs/nuttxfs_page_1.png b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_1.png new file mode 100644 index 00000000000..4d2351326d3 Binary files /dev/null and b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_1.png differ diff --git a/Documentation/components/filesystem/nuttxfs/nuttxfs_page_2.png b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_2.png new file mode 100644 index 00000000000..a22ae8e0c47 Binary files /dev/null and b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_2.png differ diff --git a/Documentation/components/filesystem/nuttxfs/nuttxfs_page_3.png b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_3.png new file mode 100644 index 00000000000..689d96f1d27 Binary files /dev/null and b/Documentation/components/filesystem/nuttxfs/nuttxfs_page_3.png differ diff --git a/Documentation/components/filesystem/nxffs.rst b/Documentation/components/filesystem/nxffs.rst index b3936e7106d..50acafbc9fe 100644 --- a/Documentation/components/filesystem/nxffs.rst +++ b/Documentation/components/filesystem/nxffs.rst @@ -184,3 +184,10 @@ Things to Do implementation of a proper truncate() method which should alter the size of a previously written file. There is some fragmentary logic in place but even this is conditioned out with __NO_TRUNCATE_SUPPORT__. + + +Further reference +================= + +.. toctree:: + how_nxffs_works.rst \ No newline at end of file diff --git a/Documentation/components/filesystem/nxffs/nxffs_page_0.png b/Documentation/components/filesystem/nxffs/nxffs_page_0.png new file mode 100644 index 00000000000..48e354738cf Binary files /dev/null and b/Documentation/components/filesystem/nxffs/nxffs_page_0.png differ diff --git a/Documentation/components/filesystem/nxffs/nxffs_page_1.png b/Documentation/components/filesystem/nxffs/nxffs_page_1.png new file mode 100644 index 00000000000..47db2551204 Binary files /dev/null and b/Documentation/components/filesystem/nxffs/nxffs_page_1.png differ diff --git a/Documentation/components/filesystem/nxffs/nxffs_page_2.png b/Documentation/components/filesystem/nxffs/nxffs_page_2.png new file mode 100644 index 00000000000..ade6107f2b1 Binary files /dev/null and b/Documentation/components/filesystem/nxffs/nxffs_page_2.png differ diff --git a/Documentation/components/filesystem/nxflat.rst b/Documentation/components/filesystem/nxflat.rst new file mode 100644 index 00000000000..6d0b2206295 --- /dev/null +++ b/Documentation/components/filesystem/nxflat.rst @@ -0,0 +1,180 @@ +================================= +NuttX FLAT Binary Format (NXFLAT) +================================= + +.. warning:: + Migrated from: + https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139630111 + +Overview +======== + +NuttX supports a configurable :doc:`binary loader <../binfmt>` . +This binary loader supports loading and executing binary objects from +the file system. +The NuttX binary loader is capable of supporting multiple binary formats. +One of of those binary formats is NXFLAT, the top of this Wiki page. + +NXFLAT is a customized and simplified version of binary format +implemented a few years ago called XFLAT. +With the NXFLAT binary format you will be able to do the following: + +* Place separately linked programs in a file system, and + +* Execute those programs by dynamically linking them to the + base NuttX code. + +This allows you to extend the NuttX base code after it has been +written into FLASH. +One motivation for implementing NXFLAT is support clean CGI under +an HTTPD server. + +This feature is especially attractive when combined with the +NuttX ROMFS support: +ROMFS allows you to execute programs in place (XIP) in flash +without copying anything other than the .data section to RAM. +In fact, the initial NXFLAT release only worked on ROMFS. +Later extensions also support execution NXFLAT binaries from +an SRAM copy as well. + +This NuttX feature includes: + +* A dynamic loader that is built into the NuttX core (See SVN). +* Minor changes to RTOS to support position independent code, and +* A linker to bind ELF binaries to produce the + NXFLAT binary format (See SVN). + +Toolchain Compatibility Problem +=============================== + +Description +----------- + +NXFLAT flat requires a specific kind of position independence. +The ARM family of GCC toolchains has historically supported +this method of position independence: All code addresses are +accessed relative to the Program Counter (PC) and a special, +`PIC register` (usually ``r10``) is used to access all data. To +load or store a data value, the contents of ``r10``, the PIC +base, is added to a constant, position-independent offset +to produce the absolute address of the data. + +.. image:: nxflat/nxflat-addressing.png + +The `Global Offset Table` (GOT) is a special data structure that +resides in D-Space. So PIC-base relative addressing may also +be specified as GOT-Relative addressing (or ``GOTOFF``). The older +GCC 4.3.3 GCC compiler, for example, generates ``GOTOFF`` +relocations to the constant strings, like: + +.. code-block:: asm + + .L3: + .word .LC0(GOTOFF) + .word .LC1(GOTOFF) + .word .LC2(GOTOFF) + .word .LC3(GOTOFF) + .word .LC4(GOTOFF) + +Where ``.LC0``, ``.LC1``, ``.LC2``, ``.LC3``, and ``.LC4`` are the labels +corresponding to strings in the ``.rodata.str1.1`` section. One +consequence of this is that ``.rodata`` must reside in D-Space +since it will addressed relative to the GOT (see the section +entitled "Read-Only Data in RAM" here). + +The newer 4.6.3 GCC compiler, however, generated PC relative +relocations to these same strings: + +.. code-block:: + + .L2: + .word .LC0-(.LPIC0+4) + .word .LC1-(.LPIC1+4) + .word .LC2-(.LPIC2+4) + .word .LC3-(.LPIC4+4) + .word .LC4-(.LPIC5+4) + +These are `PC-Relative` relocations. This means that the string +data is address not by an offset relative to the PIC register +(``r10``) but to the program count (PC). This is good and bad. This +is good because it means that ``.rodata.str1.1`` must now can now +reside in FLASH with ``.text`` and can be accessed using PC-relative +addressing. That can be accomplished by simply moving the ``.rodata`` +from the ``.data`` section to the ``.text`` section in the linker script. +The NXFLAT linker script is located at +``nuttx/binfmt/libnxflat/gnu-nxflat-?.ld``. **NOTE**: There are two +linker scripts located at ``nuttx/binfmt/libnxflat/``: + +1. ``binfmt/libnxflat/gnu-nxflat-gotoff.ld.`` Older versions + of GCC (at least up to GCC 4.3.3), use GOT-relative + addressing to access RO data. In that case, read-only + data (``.rodata``) must reside in D-Space and this linker + script should be used. +2. ``binfmt/libnxflat/gnu-nxflat-pcrel.ld.`` Newer versions + of GCC (at least as of GCC 4.6.3), use PC-relative + addressing to access RO data. In that case, read-only + data (``.rodata``) must reside in I-Space and this linker + script should be used. + +But this is very bad because a lot of NXFLAT is now broken. +For it appears that not just constant strings, but that all +data may now be referenced using PC-relative addressing to +.bss and .data values. I do not yet know the extent of this +problem or the direction that GCC is going but certainly +version 4.6.3 cannot be used with NXFLAT. + +The workaround for now is to use the older, 4.3.3 OABI +compiler. In the long run, this might spell the end +to NXFLAT. + +Update: Restored GCC Support +---------------------------- + +This post was pointed out by Michael Jung: + +.. code-block:: bash + + MCU: STM32F4 (ARM Cortex M4) + Build environment: arm-none-eabi-gcc 4.8.4 20140725 + + My goal is to build an image that can be run from any properly-aligned + offset in internal flash (i.e., position-independent). I found the + following set of gcc flags that achieves this goal: + + # Generate position independent code. + -fPIC + + # Access bss via the GOT. + -mno-pic-data-is-text-relative + + # GOT is not PC-relative; store GOT location in a register. + -msingle-pic-base + + # Store GOT location in r9. + -mpic-register=r9 + +Reference: https://gcc.gnu.org/ml/gcc-help/2015-07/msg00027.html + +Michael has verified that ``-mno-pic-data-is-text-relative`` is, +indeed, a solution to the above NXFLAT problem in newer +compilers. You simply need to modify the board Make.defs +file like: + +1. ARCHPICFLAGS = -fpic -msingle-pic-base -mpic-register=r10 + +.. code-block:: bash + + +ARCHPICFLAGS = -fpic -msingle-pic-base -mpic-register=r10 -mno-pic-data-is-text-relative + +NOTE the minor difference from the post: NuttX uses ``r10`` as +the PIC base register by default in all configurations. + +See this `thread `_ for additional information. + +References +---------- + +* :doc:`NXFLAT <../nxflat>` +* `XFLATFLAT `_ +* `FLAT `_ +* `ROMFS `_ \ No newline at end of file diff --git a/Documentation/components/filesystem/nxflat/nxflat-addressing.png b/Documentation/components/filesystem/nxflat/nxflat-addressing.png new file mode 100644 index 00000000000..e3a4764e280 Binary files /dev/null and b/Documentation/components/filesystem/nxflat/nxflat-addressing.png differ diff --git a/Documentation/components/filesystem/pseudofs.rst b/Documentation/components/filesystem/pseudofs.rst new file mode 100644 index 00000000000..8f716ada971 --- /dev/null +++ b/Documentation/components/filesystem/pseudofs.rst @@ -0,0 +1,211 @@ +================== +Pseudo File System +================== + +.. warning:: + Migrated from: + https://cwiki.apache.org/confluence/display/NUTTX/Pseudo+File+System + +Overview +======== + +Pseudo Root File System +----------------------- + +NuttX includes an optional, scalable file system. As a minimum, this may +be a simple in-memory, pseudo file system. This is an in-memory file +system because it does not require any storage medium or block driver +support. Rather, file system contents are generated on-the-fly as +referenced via standard file system operations (open, close, read, +write, etc.). In this sense, the file system is a pseudo file system +(in the same sense that the Linux ``/proc`` file system is also referred +to as a pseudo file system). + +Any user supplied data or logic can be accessed via the pseudo-file +system. Built in support is provided for character, block, and MTD +(Memory Technology Device) drivers in the ``/dev`` pseudo file system +directory. + +Special Files +------------- + +NuttX does not support special files in the way that, say, +Linux does. In fact, it is more correct to say that NuttX +file systems do not support special files at all. + +NuttX does, however, support Linux-like special `device node`, +character driver, and block driver files (as well as NuttX-specific +mountpoint, named semaphore, message queue, and shared memory +special files). However, these are not special files in sense +that the term special files is used in a POSIX environment: In +NuttX these special files may `only` be created in the root +pseudo-file system. For the case of device nodes, see `Device +Nodes `_ +for further information. + +In NuttX, the underlying principle is that all `named resources` +appear as special files in the root pseudo-file system and are +managed by the VFS. + +Mounted Volumes +--------------- + +The simple in-memory file system can be extended by mounting block +devices that provide access to true file systems backed up via +some mass storage device. NuttX supports the standard mount() +command that allows a block driver to be bound to a mount point +within the pseudo file system and to a file system. At present, +NuttX supports the standard VFAT and ROMFS file systems, a +special, wear-levelling NuttX FLASH File System (NXFFS), as well +as a Network File System client (NFS version 3, UDP). + +Comparison to Linux +------------------- + +From a programming perspective, the NuttX file system appears +very similar to a Linux file system. However, there is a +fundamental difference: The NuttX root file system is a pseudo +file system and true file systems may be mounted in the pseudo +file system. In the typical Linux installation by comparison, +the Linux root file system is a true file system and pseudo +file systems may be mounted in the true, root file system. +The approach selected by NuttX is intended to support greater +scalability from the very tiny platform to the moderate platform. + +FAQ +=== + +**Question**: I'm wondering why I can't create a directory. If +I try to create a dir. + +.. code-block:: bash + + mkdir /mnt + +I get this, + +.. code-block:: bash + + nsh: mkdir: mkdir failed: 2 + +although if I do this it creates both directories, mnt and sda + +.. code-block:: bash + + mount -t vfat /dev/mmcsd0 /mnt/sda + +**Answer**: This is because the top level directories are part of a +`pseudo-filesystem` – like the Linux ``proc/`` or ``sys/`` file systems. +But the NuttX pseudo-file system begins at the top level ``/``. + +What that really means is that you do must have +``CONFIG_DISABLE_PSEUDOFS_OPERATIONS`` selected. Because you +can normally create directories in the pseudo-filesystem +with not problem: + +.. code-block:: bash + + NuttShell (NSH) NuttX-9.0.0 + nsh> mkdir /mnt + nsh> ls + /: + dev/ + etc/ + mnt/ + proc/ + tmp/ + nsh> ls mnt + /mnt: + nsh> + +But lets assume that you do have operations on the pseudo-file +system disabled. Why doesn't it work? There is no `real` media +there so you cannot create a file there or create any directories +there. The ``mount`` command is special, it knows how to create mount +points in the pseudo-file system. + +The pseudo-file system is just a tree structure in RAM. +It serves two purposes: (1) you don't have to have a real +file system to use NuttX. +It comes up out-of-the-box with usable (but limited) +pseudo-file system. That allows a little more civilized +programming environment on even very resource limited MCUs. +And (2) this pseudo-file system is a place where all special +NuttX files are retained: Character drivers, block drivers, +and mount points. + +The NuttX top-level pseudo-filesystem creates the `illusion` of +directories and provides a consistent, seamless semantic for +interacting with mounted file systems. If there is a file +called ``hello.txt`` in your volume mounted at ``/mnt/sda``, then: + +``/mnt`` - is a `node` in the pseudo-filesystem that does +nothing but contain the name mnt and provide links +to things `under` ``mnt``. + +``/mnt/sda`` - This refers to a node that contains the name +sda that can be found `under` the node with the name mnt. +This node is a special `mountpoint node` in the pseudo-filesystem. +It contains the methods needed to interact will real file system. +Everything `below` ``/mnt/sda`` is in the physical media. + +``/mnt/sda/hello.txt`` - This, then refers to the file +``hello.txt`` at the relative path ``hello.txt`` on the mounted media. +The transition from the pseudo-filesystem to the +real media is seamless. + +This is a little different from Linux: Linux always +has to boot up with a `real` file system – even if it +is only a initrd RAM disk. +In Linux, these special files (links, drivers, pipes, +etc.) reside on real media and can reside in any +Linux-compatible filesystem. + +Normal ``mkdir`` can only work if there is a `real` filesystem +at the location. There are no real directories in the +pseudo-filesystem. The pseudo-filesystem does support +`nodes` that look like directories and have some of the +properties of directories (like the node ``/mnt`` mentioned +above). But this is really an illusion. + +If ``CONFIG_DISABLE_PSEUDOFS_OPERATIONS`` is not enabled, +then NuttX adds the capability to create new, empty `nodes` +in the pseudo-filesystem using ``mkdir``, completing the illusion. + +[On the other hand, all directories are really an +`illusion` in a way and I suppose that in that sense +these nodes the pseudo-filesystem are just as `real` +as any other directory.] + +After you mount the SD card at ``/mnt/sda``, then you can do: + +.. code-block:: bash + + mkdir /mnt/sda/newdir + +That should work fine and should create a directory at +the relative path ``newdir`` in the mounted volume. + +There are a few other special NSH commands like mount that +can change the pseudo-filesystem. Like ``losetup``, ``mkfifo``, +``mkrd``, ``umount``, etc. +In fact, these commands `only` work in the pseudo-filesystem. +Try them in ``/mnt/sda``... they won't work. + +But none of the `normal` commands that modify files or directories +will work in the pseudo-filesystem: ``mkdir``, ``mv``, ``rm``, ``rmdir``. +These all require real media. They will not work in the +pseudo-filesystem, but will work in ``/mnt/sda``. + +And trying to pipe to something in the pseudo-filesystem +will also fail. You cannot do this, for example: + +.. code-block:: bash + + NuttShell (NSH) NuttX-6.20 + nsh> cat "Hello, World!" >/hello.text + nsh: cat: open failed: 22 + nsh> + +See also NxFileSystem in +`Porting Guide `_ \ No newline at end of file diff --git a/Documentation/components/filesystem/special_files_dev_num.rst b/Documentation/components/filesystem/special_files_dev_num.rst new file mode 100644 index 00000000000..2e0fd615f62 --- /dev/null +++ b/Documentation/components/filesystem/special_files_dev_num.rst @@ -0,0 +1,122 @@ +================================ +Special Files and Device Numbers +================================ + +.. warning:: + Migrated from: + https://cwiki.apache.org/confluence/display/NUTTX/Special+Files+and+Device+Numbers + +Special Files in Unix-like File Systems +======================================= + +In Unix-like operating systems, a `special file` is an interface for +a device driver that appears in a file system as if it were an +ordinary file. Special files are a feature built into Unix-like +file systems such as EXT2. + +See also `Wikipedia article `_. + + +Special Files in Other File Systems +=================================== + +Some other file systems systems, such as NTFS, also have some more +limited and incompatible notion of special files. Others, such as +FAT, do not have any such concept. Unix-like environments such as +Cygwin that run on these other file systems have to do things a +little differently: The have to emulate special files using the +resources of the file system that they operate on. This may mean +creating `regular` files with special naming conventions or with +special content that can be used to emulate the behavior of +special files. + +Special Files and NuttX +======================= + +NuttX has done things in a very different way. There are no +special files supported in any file system. Rather, special +files can exist only in the NuttX :doc:`pseudo file system `. This +was a decision that was made in the initial design to simplify +things for resource limited platforms yet still provide a mostly +standard Unix-like/POSIX programming environment. + +What are the advantages of the special files in the NuttX +pseudo-file system? Reduce resource usage, reduced bring-up +requirements. What are the disadvantages? In NuttX, special +files can only reside in the pseudo-file system. + +The only other consequence that I aware of is that NuttX cannot +support the POSIX requirement for the ``st_dev`` field in the +``struct stat`` structure. + + +Device Files and Device Numbers +=============================== + +In a Unix-like system, devices are access special device files. +In a Unix-like system, device files can reside in any compatible +file system but, by convention, are always placed in the ``/dev`` +directory. NuttX achieves programming compatibility with this +convention because the top-level, `root` file system is the +pseudo-file system and ``/dev`` is part of the pseudo-file system. + +But there is a bigger difference that this. The bigger +difference is how device drivers are registered and how +the are accessed. The primary content of the Unix-like +device file is simply a number, a device number, usually +represented as type ``dev_t``. The device number an encoded +that consists of a `major` device number and a `minor` device +number. The major device number identifies the type of +driver and the minor number identifies an instance of a +driver of that type. + +There is nothing special about these number from the sense of +a file system. Just because device file exists with a +certain major and minor number, that does not mean that there +is actually any real driver instance backing that device file +up. For example, you can create a device file from the Linux +command line like this, knowing nothing other than major and +minor device number: + +.. code-block:: C + + dev_t makedev(unsigned int maj, unsigned int min); + +In a Unix-like system, when you try to open a device driver +several things must happen: The system must open the device +file, obtain the device major and minor number, and then +look up the driver instance in some internal `registry` of +registered device drivers. If one is found, then the system +can complete the open operation. + +So, the device number is then a `key` of some kind into a +`registry` of device drivers. NuttX does this very differently: +There are no device numbers, rather the pseudo-file system `is` +the device registry! In NuttX, device files cannot be created +by users; they can only be created by device drivers by calling +the following, internal interface: + +.. code-block:: c + + int register_driver(FAR const char *path, FAR const struct file_operations *fops, mode_t mode, FAR void *priv); + +The ``path`` argument determines where in the pseudo-file system the +device file should be placed. ``mode`` provides device file privileges. +The ``fops`` and ``priv`` provide the internal information for the ``registry``. + +So the when you open a device driver in NuttX, many fewer steps are +involved: The system must still open the device file, but then +since the pseudo-file system `is` the device registry, all of the +device driver information is available and ``open`` operation +completes with no further actions. + +Named Resources +=============== + +This use of the pseudo-file system in NuttX to manage device +files is consistent with a core NuttX device philosophy: The +NuttX VFS and the pseudo-file system in particular, are used +to manage all named OS resources. That applies not only to +device files and other special files but also to such things +named message queues and named semaphores (which can be found +in the pseudo-file system in the ``/var`` directory). \ No newline at end of file