ROMFS is the usual way to carry executables on a NOMMU target with memory
mapped NOR flash: it can hand out a real flash pointer from mmap(), so the
NXFLAT loader maps a module's text in place instead of copying it into RAM.
But a ROMFS image is built on the host and is read only, so a module cannot
be downloaded onto the board at run time.
xipfs is a writable file system with the same in-place property. Each file
is stored as one physically contiguous, erase-block aligned extent, so an
mmap() of it resolves to flash_base + extent_offset and a loader can execute
the file where it already lies. This needs the underlying MTD driver to
answer BIOC_XIPBASE; on the RP2350 rp23xx_flash_mtd.c does.
Files are write once. A file is created, its size is declared, it is
written sequentially, closed, and is thereafter immutable until it is
deleted. That is the whole life cycle of a downloaded module, and it is
what licenses the design: the exact extent is reserved at create time, so
no file ever grows, moves, or fragments internally. Random writes, appends
and truncation of a written file are not supported and are refused.
The only source of fragmentation is therefore free space holes left by
deletes. Allocation fails with -ENOSPC when no single contiguous run is
large enough, and never defragments on its own; the caller decides whether
to compact and retry, through XIPFSIOC_DEFRAG. Defragmentation is manual,
best effort and interruptible: it is a loop of atomic single-extent
relocations, each one copy, commit, erase, so every stop point -- a time
budget, a pinned extent, an erase error -- leaves a consistent layout that
is simply less compact. It reports the largest contiguous run it achieved,
which is what tells the caller whether the retry will fit.
Metadata is committed power safely. Two metadata block sets are used in
ping-pong, each generation carrying a sequence number and a CRC, and every
state change is ordered as write the new data, flip the metadata reference,
then erase what the old one referenced. Mount scans both sets and selects
the last fully valid generation, so a torn write costs the interrupted
operation and nothing else.
A mapping takes a pin on the extent, and the pin lives on the extent rather
than on the file descriptor, so three running instances of one module hold
three pins and the extent becomes movable only when the last one goes.
Defragmentation skips pinned extents, which is what stops it relocating
code that is executing. The pin is released by munmap() or by the task
teardown walk, so a task that dies without unmapping does not leak it.
Directories are records in that same generation, carrying their own identity
and the identity of the directory holding them; the root is implicit and owns
identity zero. They are deliberately NOT objects in the data region, which
is what keeps the commit story in one piece: mkdir and rmdir add or remove a
record and commit one generation, exactly as create and unlink do, so there
is never a multi-object update to journal or an orphan to collect at mount.
An empty directory therefore exists, survives a remount, and costs one entry
out of the volume's fixed supply and no flash blocks at all.
A name is one path component; depth comes from the parent, so XIPFS_NAME_MAX
bounds a component, which is what statfs reports it as. Mount rebuilds the
tree and checks that it is one: identities unique, names unique within a
directory, every parent a live directory, and following parents reaching the
root -- a cycle on the medium would otherwise hang a path walk rather than
merely answering wrongly. '.' and '..' are refused as components, since an
entry stored under either could never be reached again.
The commands that act on the volume rather than on one file --
XIPFSIOC_DEFRAG and XIPFSIOC_LISTPINNED -- are reached through the ioctldir
method, on a descriptor for the mountpoint directory. They are accepted on
a descriptor for a file inside the volume too, but that route holds the file
open for the duration and an open extent cannot be relocated, so a pass
asked for that way is obstructed by the act of asking.
mmap() falls back to the generic RAM copy for ordinary readers when the
media cannot be addressed directly. A module loader must not silently get
a RAM copy, so MAP_XIP_STRICT is added: with it the mapping either resolves
in place or fails with -ENXIO, which the caller can turn into defragment
and retry.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
During the Toybox port to NuttX, Claude noticed that changes in the
menuconfig weren't taking affect. This issue exists for a long time on
NuttX, in fact BayLibre's presentation from 2017 make jokes about our
building system not been reliable:
https://www.youtube.com/watch?v=XUJK2htXxKw&t=320s
Stale archive members from $(AR)'s additive-only behavior can linger
after Kconfig toggles change which files provide a symbol, causing dead
weight or "multiple definition" link errors on incremental builds.
Fixed by splitting ARCHIVE into two macros: ARCHIVE keeps the original
additive behavior for apps/libapps.a, which many independent
subdirectories contribute to across a build, while the new
ARCHIVE_REBUILD deletes then archives for the far more common case
of a single Makefile building its own self-contained $(OBJS)
- all 39 such call sites now use it.
Assisted-By: Claude Sonnet 5
Signed-off-by: Alan C. Assis <acassis@gmail.com>
Intermediate files of make depend like .ddc and .dds may remain
when make is interrupted. Remove them using make distclean.
Signed-off-by: SPRESENSE <41312067+SPRESENSE@users.noreply.github.com>
Most tools used for compliance and SBOM generation use SPDX identifiers
This change brings us a step closer to an easy SBOM generation.
Signed-off-by: Alin Jerpelea <alin.jerpelea@sony.com>
This extension for the event group, to establish a connection between
named event groups and a task. The task may reference the event group
associated with a VFS-based name. Then the event group can be used
in subsequent calls to nxevent_wait() or nxevent_post().
This is an internal OS interface and should not be used by applications.
Signed-off-by: fangxinyong <fangxinyong@xiaomi.com>
Summary:
1.Add configuration to allocate memory from the specified section
2.Replace all memory operations (kmm_) in the vfs with
fs_heap_. When FS_HEAPSIZE > 0, memory is requested for the file system by specifying a configured heap location. By default (i.e. FS_HEAPSIZE=0) fs_heap_ is equivalent to kmm_
Signed-off-by: chenrun1 <chenrun1@xiaomi.com>
Revert "Parallelize depend file generation"
This reverts commit d5b6ec450f.
parallel depend ddc does not significantly speed up compilation,
intermediately generated .ddc files can cause problems if compilation is interrupted unexpectedly
Signed-off-by: xuxin19 <xuxin19@xiaomi.com>
For some case, need large files from tmpfs or shmfs, will lead to high
pressure on memory fragments, add an optional fs_heap with independent
heap will benifit for memory fragments issue.
Signed-off-by: buxiasen <buxiasen@xiaomi.com>
we can mount a zipfile in nuttx use mount command like this:
mount -t zipfs -o /data/test.zip /zip
The zipfs is a read only file system,The advantage is that it
does not occupy additional space when reading the decompressed file.
When used, reading and decompression operations are simultaneous.
The known disadvantage is that when using seek to read forward,
it will reopen and cause slow speed problems.
Signed-off-by: anjiahao <anjiahao@xiaomi.com>
In the past, header file paths were generated by the incdir command
Now they are generated by concatenating environment variables
In this way, when executing makefile, no shell command will be executed,
it will improve the speed of executing makfile
Signed-off-by: yinshengkai <yinshengkai@xiaomi.com>
Make.dep file should be updated by .config changed after first make.
There are 2 cases affected for this problem:
1) Add source files by config symbol
2) Include header files in #ifdef directive
These 2 cases may not be included in Make.dep and this may prevent the
differential build from working correctly.
Since littlefs is in active development, it's not a good idea to use its
source code files directly. So upgrade littlefs v2.2.1 by using the littlefs
tar.gz release package instead.
Change-Id: I16d9cf0b6bca700a54ca86ed11d7c8c7f27a898f
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
Squashed commit of the following:
configs/: The few configurations that formerly set CONFIG_NFILE_DESCRIPTORS=0 should not default, rather they should set the number of descriptors to 3.
fs/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
tools/: Tools updates for changes to usage of CONFIG_NFILE_DESCRIPTORS.
syscall/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
libs/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
include/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
drivers/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
Documentation/: Remove all references to CONFIG_NFILE_DESCRIPTORS == 0
binfmt/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
arch/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
net/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
sched/: Remove all conditional logic based on CONFIG_NFILE_DESCRIPTORS == 0
sched/Kconfig: CONFIG_NFILE_DESCRIPTORS may no longer to set to a value less than 3
configs/: Remove all settings for CONFIG_NFILE_DESCRIPTORS < 3
Squashed commit of the following:
Author: Gregory Nutt <gnutt@nuttx.org>
fs/littlefs: Fix some compilation errors.
fs/littlefs: Brings code a little closer to NuttX coding standard. Not 100%, but closer.
fs/littlefs: Convert all C++ style comments to C comments.
Author: lihaichen <li8303@163.com>
fs/littlefs: Adds port of the mbed littlefs.
depends on !DISABLE_MOUNTPOINT && MTD_BYTE_WRITE
register_mtddriver("/dev/w25", mtd, 0755, NULL);
mount("/dev/w25", "/w25", "littlefs", 0, NULL);
Squashed commit of the following:
fs/spiffs: Fix last compilation issue. Now compiles without error. It is still not quite ready for testing as there is additional code review that must be be performed. It is now marked as EXPERIMENTAL so that it can be brought onto the master branch with little risk.
fs/spiffs: Remove some dead code.
fs/spiffs: Weak start of analysis of spiffs_nucleus.c. Renamed to spiffs_core.c
fs/spiffs: Rename spiffs_nucleus.c to spiffs_core.c
fs/spiffs: Remove spiffs_config.h. All configuration settings are now available in the SPIFFS Kconfig options.
fs/spiffs: Finished review, update, and repartitioning of spiffs_check.c. Added spiffs_check.h.
fs/spiffs: Finished review, update, and repartitioning of spiffs_cache.c. Added spiffs_cache.h.
fs/spiffs: Clean up some defines used in debug output statements.
fs/spiffs: Finished review, update, and repartitioning of spiffs_gc.c. Added spiffs_gc.h.
fs/spiffs: Now that VFS interface is completed, I have begun the long march of repartitioning the remaining functionality, reviewing logic, identifying dead code, and cleaning up loose ends.
fs/spiffs: Initial integration of MTD interface, replacing the SPIFFS native flash interface. Lots of open issues such as the use of pages vs. blocks vs. erase blocks and units of addresses, offsets, and lengths that are passed in function calls. Remove SPIFFS_USE_MAGIC support. That option (which default to OFF anyway), wrote a magic value at the beginning of every sector and support verifiable identification of the file system. It was not being and used and removing it makes life simpler.
fs/spiffs: Remove semaphore lock on the file object structure. Ultimately, the file access must modify the volume and access the volume structue which also has a exclusivity lock. So use of the volume lock alone should be sufficient.
Integrated the SPIFFS rename logic into the NuttX VFS. Removed non-standard application calls or convert them to IOCTL commands. These were converted to IOCTL commands: (1) integrity check, (2) garbage collection, and (3) format flash. These were removed: (1) Integrity check callback. These provided a lot of good information about the state of the file system, but such callbacks are not compatible with a POSIX compliant file system. (2) Index maps. The index maps were a performance improvement feature. The user could provide the memory and request that a region of a a file use that memory for improved lookup performance when accessing parts of the file. The fallback is the less performance lookup by traversing the FLASH memory. (3) Removed the quick garbage collection interface (the code is still used internally). Only the full garbage collection is available to the user application via IOCTL.
configs/sim/spiffs: A simulator configuration to use for testing SPIFFS.
fs/spiffs: Integrate SPIFFS logic into NuttX VFS bind() and unbind() methods.
fs/mount/fs_mount.c: Add SPIFFS to the list of drivers that require MTD vs block drivers.
fs/spiffs: Trivial changes, mostly from analysis of how to integrate the rename() VFS method.
fs/spiffs: Connect NuttX VFS unlink method to the SPIFFS_remove() function. Lots of name-changing.
fs/spiffs: Remove non-standard errno support. Remove bogus SPIFFS_LOCK() and SPIFFS_UNLOCK() macros.
fs/spiffs: Add NuttX VFS implementation for statfs() method. Clean up some of the accumulating compilation problems.
fs/spiffs: Add stat(), truncate() methods. Dummy out unsupport mkdir() and rmdir() methods.
fs/spiffs: Replace some of the custom error numbers with standard error numbers.
fs/spiffs: Hooks read(), write(), fstat(), ioctl(), opendir(), closedir(), rewindif(), and readdir() into the NuttX VFS.
fs/spiffs: Beginning the organization to work with the NuttX VFS. Lots of things are get broken!
fs/spiffs: Add spiffs.c which will be the interface between SPIFFS and NuttX. No very close at present, however.
fs/spiffs: Clean up some compile problems introduced by coding standard changes.
fs/spiffs: A little closer to NuttX coding standard.
fs/spiffs: Ran tools/indent.sh against all files. Closer to NuttX coding standard, but needs a lot more effort to be fully compliant.
fs/spiffs: This commit brings in version 0.3.7 of Peter Anderson's SPIFFS. The initial commit includes the core FS files (with some definitions destributed to their correct header files) and hooks into the build system.
Squashed commit of the following:
fs/cromfs: Now depends on EXPERIMENTAL because it is not yet verified.
fs/cromfs: This commit brings the CROMFS file system to code complete. Still completely untested. Next steps: Need a tool to generate CROMFS file system images and a test case under apps/
fs/cromfs: Add logic to traverse the in-memory file system nodes.
fs/cromfs: Add initial support for an in-memory, compressed, read-only file system.
fs/userfs: This completes coding of the UserFS client and of the UserFS feature in general. This feature is being merged to main now because I believe it is innocuous. It is, however, untesed. The next step will be to develop a test case to verify the feature.
fs/userfs: Completes the request logic for the UserFS client. Still need the logic that receives the responses.
fs/userfs: Completes coding for most of the server side of the user filesystem logic.
fs/userfs: Big design changes, simplications. Use Unix domain local sockets instead of message queues. Easier to transfer big data in local sockets than message queues. Remove character drvier 'factory' it is not necessary.
fs/userfs: Minor reparitioning; volume private info does not need to be held on the OS client side.
libc/userfs: Add some of the server side logic.
fs/userfs: Add some UserFS initialization logic.
fs/userfs: Add frame work for the UserFS proxy. Remove all references to a block driver. There is no block dricer... what was I thinking?
fs/userfs: Add some initialization of the character driver, 'factory' device.
fs/userfs: Rename from fusefs to userfs to that we don't stomp on someone else's cool name.
Add a header file describing the fusefs interface.
