memcmp, strncmp and strcmp reach their word loops only when both pointers
are already on a register boundary:
or t0, a0, a1
andi t0, t0, SZREG-1
That asks more than the loops need. They load from the two pointers at
the same boundary, so what matters is that the two agree about where a
boundary falls, not that either is already on one. A pair offset by the
same amount can be walked up to the boundary a byte at a time and
compared a register at a time from there.
The union also holds far less often than the difference. For arbitrary
pointers on RV64 it is true about one time in 64 against one in eight,
and the case it rejects, two strings carved out of the same buffer, is
the common one.
Test the difference of the pointers, and walk to the boundary first.
arch_strcpy.S and arch_memcpy.S already do this. Keeping every access
aligned is not only faster here: the base ISA does not require misaligned
loads and stores to be supported at all, so a routine in a machine
directory cannot assume one will work, whatever it costs.
Measured on a 1.4 GHz rv64, source and destination misaligned by one:
before after
memcmp 32K 34.4 458.0 MB/s
strncmp 32K 32.4 253.0 MB/s
strcmp 32K 41.0 280.0 MB/s
Each of those was the rate of the byte loop the word loop was meant to
replace. Pointers that genuinely disagree still take the byte loop, and
the aligned rates are unchanged.
The measurements come from the benchmark in apache/nuttx-apps#3706.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
The word loop walks src to a register boundary and then stores a whole
register at a time to dst, but nothing establishes that dst is on a
boundary too. Where the two pointers disagree about where a boundary
falls, every store in that loop is misaligned.
The base ISA does not require misaligned stores to be supported. Where
firmware emulates them each store traps into machine mode, and where
nothing emulates them the store faults, so this is not only a question of
speed. Measured on a 1.4 GHz rv64 that emulates them, with a 32 KB
string whose src and dst are misaligned by different amounts:
generic C 410.4 MB/s
this file 7.5 MB/s
which is around 178 cycles per byte, flat from 512 bytes to 32 KB.
Test the two pointers against each other before going wide, as
arch_strcpy.S already does. Pointers that agree still reach the word
loop, since walking src to a boundary walks dst to one as well; pointers
that disagree take the byte path, where no single boundary serves both.
After the change the misaligned case runs at 490 MB/s and the aligned
rates are unchanged.
The measurements come from the benchmark in apache/nuttx-apps#3706.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
Rewrite arch_memcpy.S and arch_memset.S to be register-width aware on
both RV32 and RV64 using REG_L/REG_S/SZREG macros from asm.h.
memcpy gains:
- 16xSZREG unrolled main loop (128B/iter on RV64, 64B on RV32).
- Shift-merge path for misaligned src: reads two aligned words
straddling each output word and shifts them together, so no load
or store is ever misaligned.
- Single SZREG and byte loops for remainder and small copies.
memset gains:
- 32xSZREG unrolled main loop (256B/iter on RV64, 128B on RV32)
using Duff's device for non-power-of-two remainders.
- .option norvc ensures fixed 4-byte instruction width for correct
jump offset calculation in the Duff's device entry.
- Zero-length input handled correctly (branch to guarded tail).
The old memcpy always used lw/sw even on RV64, wasting half the
memory bandwidth. The old memset unrolled only 16 bytes per iteration.
Signed-off-by: ganjing <ganjing@xiaomi.com>
Add word-at-a-time strlcpy using DETECTNULL for both the copy phase
and the strlen tail when truncated. The copy loop aligns src and
processes a register at a time, falling to bytewise for the last word
containing the terminator. When truncated, the remaining src length
is measured with a second word-at-a-time loop.
strlcpy has 46 call sites in a typical kernel image (more than strcpy)
and is not covered by newlib OPTSPEED, making it a high-value target.
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
Reduce branch overhead in the memcmp main loop by comparing four
words per iteration: XOR each pair, OR the four differences together,
and branch once. On a mismatch the single-word loop locates the
exact differing word within four words of the fault.
Add a beqz guard at .Lbyte_cmp entry to handle the case where the
4-word loop consumes all remaining bytes exactly.
Measured on QEMU RV32: memcmp(128) 313 -> 271 cycles (13% faster).
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
Add assembly-optimized implementations for 14 string/memory functions
using word-at-a-time techniques (DETECTNULL, broadcast+XOR) and
XLEN-adaptive macros for both RV32 and RV64:
- memmove: direction check + forward tail to memcpy, reverse path
with 16xSZREG unroll and shift-merge for misaligned src.
- memcmp: word-granularity compare when both pointers share alignment,
bytewise fallback for mismatched pointers.
- memchr: broadcast target byte, XOR with each word, DETECTNULL to
find matches. Counter-based bounds (no pointer overflow).
- strlen: DETECTNULL word loop, constants loaded from .srodata.
- strnlen: strlen with counter-based length limit.
- strcpy/strncpy: word loop with DETECTNULL, zero-fill remainder
for strncpy. strncpy reuses strcpy via #define USE_AS_STRNCPY.
- stpcpy/stpncpy: reuse strcpy/strncpy via #define USE_AS_STPCPY.
- strchr/strchrnul: broadcast+XOR detecting both target char and
null simultaneously. strchrnul reuses strchr via #define.
- strrchr: forward scan recording last match position.
- strncmp: word-at-a-time compare with null detection and counter.
- strcat: strlen(dst) then strcpy(dst_end, src) word-at-a-time.
Each function is independently selectable via CONFIG_RISCV_<FUNC>,
or all enabled together with CONFIG_RISCV_STRING_FUNCTION=y.
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
debug.h is a NuttX-specific, non-POSIX header. Placing it in the
top-level include/ directory creates naming conflicts with external
projects that define their own debug.h.
This commit moves the canonical header to include/nuttx/debug.h,
following the NuttX convention for non-POSIX/non-standard headers,
and updates all in-tree references.
A backward-compatibility shim is left at include/debug.h that
emits a deprecation #warning and re-includes <nuttx/debug.h>,
allowing out-of-tree code to continue building while migrating.
Signed-off-by: Piyush Patle <piyushpatle228@gmail.com>
(gdb) bt
at /home/mi/ssd/dev-system/nuttx/include/string.h:321
ppcmd=<synthetic pointer>, vtbl=0x80069498) at nsh_parse.c:1909
param=0x800692b8) at nsh_parse.c:2593
cmdline=cmdline@entry=0x80069750 "hello") at nsh_parse.c:3028
argc=argc@entry=1, argv=argv@entry=0x80068870) at nsh_session.c:246
at nsh_consolemain.c:75
at nsh_main.c:74
...
Signed-off-by: yangao1 <yangao1@xiaomi.com>
riscv-none-elf-ld: /home/ligd/platform/dev-system/nuttx/staging/libc.a(arch_libc.o): in function `memcpy':
/home/ligd/platform/dev-system/nuttx/libs/libc/machine/arch_libc.c:131:(.text.memcpy+0x4e): undefined reference to `arch_memcpy'
Signed-off-by: ligd <liguiding1@xiaomi.com>
Summary:
- Replace direct use of `fence.i` instruction with `__ISB()` macro for instruction synchronization
- This change improves portability while maintaining the same functionality
Impact:
- No functional changes - both `fence.i` and `__ISB()` ensure instruction
synchronization on RISC-V
- Makes the code more maintainable by using the architecture abstraction
layer
Signed-off-by: Huang Qi <huangqi3@xiaomi.com>
kasan does not instrument the assembly function,
so it is checked in advance before calling.
If Kasan is not turned on, the speed and space will
be almost unchanged under compiler optimization.
Signed-off-by: anjiahao <anjiahao@xiaomi.com>
Most tools used for compliance and SBOM generation use SPDX identifiers
This change brings us a step closer to an easy SBOM generation.
Signed-off-by: Alin Jerpelea <alin.jerpelea@sony.com>
This applies uintreg_t in risc-v commons and fixes araised ci issues
for multiple devices. The FLAT build runs on qemu-rv64ilp32 target.
Signed-off-by: Yanfeng Liu <yfliu2008@qq.com>
As pointed out in #11322 there is a hardware design issue in RISC-V that
affects RV64 relocations. The problem is with how address bits are loaded
into registers via lui / auipc and sign extension.
If the hi20 relocation value happens to have its 32-bit sign bit set, i.e.
value is 0x80000000 (but not negative! i.e. negative in 64-bit format) the
relocation will fail, as the address is erroneously sign extended:
0x00000000_80000000 becomes 0xffffffff_80000000 which is not correct.
Also, make sure the correct opcode is used with PCREL_HI20, it expects
AUIPC (not LUI). The C compiler will never emit such code but when hand-
writing assembly code this can happen.
There is a problem with the current elf loader for risc-v: when a pair of
PCREL_HI20 / LO12 relocations are encountered, it is assumed that these
will follow each other immediately, as follows:
label:
auipc a0, %pcrel_hi(symbol) // R_RISCV_PCREL_HI20
load/store a0, %pcrel_lo(label)(a0) // R_RISCV_PCREL_LO12_I/S
With this assumption, the hi/lo relocations are both done when a hi20
relocation entry is encountered, first to the current instruction (addr)
and to the next instruction (addr + 4).
However, this assumption is wrong. There is nothing in the elf relocation
specification[1] that mandates this. Thus, the hi/lo relocation always
needs to first fixup the hi-part, and when the lo-part is encountered, it
needs to find the corresponding hi relocation entry, via the given "label".
This necessitates (re-)visiting the relocation entries for the current
section as well as looking for "label" in the symbol table.
The NuttX elf loader does not allow such operations to be done in the
machine specific part, so this patch fixes the relocation issue by
introducing an architecture specific cache for the hi20 relocation and
symbol table entries. When a lo12 relocation is encountered, the cache
can be consulted to find the hi20 part.
[1] https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-elf.adoc
Provide a way to only customize specific string operations,
such as for memcpy with the DMA capability by ROM.
Signed-off-by: yangdongdong <yangdongdong@xiaomi.com>
1. Update all CMakeLists.txt to adapt to new layout
2. Fix cmake build break
3. Update all new file license
4. Fully compatible with current compilation environment(use configure.sh or cmake as you choose)
------------------
How to test
From within nuttx/. Configure:
cmake -B build -DBOARD_CONFIG=sim/nsh -GNinja
cmake -B build -DBOARD_CONFIG=sim:nsh -GNinja
cmake -B build -DBOARD_CONFIG=sabre-6quad/smp -GNinja
cmake -B build -DBOARD_CONFIG=lm3s6965-ek/qemu-flat -GNinja
(or full path in custom board) :
cmake -B build -DBOARD_CONFIG=$PWD/boards/sim/sim/sim/configs/nsh -GNinja
This uses ninja generator (install with sudo apt install ninja-build). To build:
$ cmake --build build
menuconfig:
$ cmake --build build -t menuconfig
--------------------------
2. cmake/build: reformat the cmake style by cmake-format
https://github.com/cheshirekow/cmake_format
$ pip install cmakelang
$ for i in `find -name CMakeLists.txt`;do cmake-format $i -o $i;done
$ for i in `find -name *\.cmake`;do cmake-format $i -o $i;done
Co-authored-by: Matias N <matias@protobits.dev>
Signed-off-by: chao an <anchao@xiaomi.com>
since symbols defined in arch/elf.h is also used in other case, for example:
CC: pthread/pthread_testcancel.c machine/arm/gnu_unwind_find_exidx.c:32:8: error: unknown type name '__EIT_entry'
32 | static __EIT_entry *__exidx_start_elf;
| ^~~~~~~~~~~
Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
Add the `LIBC_PREVENT_STRING_KERNEL` and `LIBC_PREVENT_STRING_USER`
that are meant to be selected by the chip if no libc implementation
is going to be built. If selected, neither NuttX's software version
of the libc nor any architecture-specific implementation will be
built in the kernel or in the userspace, respectively. In this
case, the linker may provide a ROM-defined version of the libc
functions instead.
Summary:
- This commit adds R_RISCV_JAL support to arch_elf.c
- The code only checks the immediate value but does not relocate it
because it is done by the compiler.
Impact:
- None
Testing:
- Tested with rv-virt:knsh64 (not merged yet)
Signed-off-by: Masayuki Ishikawa <Masayuki.Ishikawa@jp.sony.com>
The jump instruction relocation had an assert that tests for jumps with
an offset of 0. This makes it so that a while(1); statement causes an
assert because the jump instruction points to the same address, which
is perfectly legal.
Addend was not handled correctly in several reloc types.
Add ADD32/64 + SUB32/64 relocations, for some reason the compiler
I use likes to add them.
