memcmp, strncmp and strcmp reach their word loops only when both pointers
are already on a register boundary:
or t0, a0, a1
andi t0, t0, SZREG-1
That asks more than the loops need. They load from the two pointers at
the same boundary, so what matters is that the two agree about where a
boundary falls, not that either is already on one. A pair offset by the
same amount can be walked up to the boundary a byte at a time and
compared a register at a time from there.
The union also holds far less often than the difference. For arbitrary
pointers on RV64 it is true about one time in 64 against one in eight,
and the case it rejects, two strings carved out of the same buffer, is
the common one.
Test the difference of the pointers, and walk to the boundary first.
arch_strcpy.S and arch_memcpy.S already do this. Keeping every access
aligned is not only faster here: the base ISA does not require misaligned
loads and stores to be supported at all, so a routine in a machine
directory cannot assume one will work, whatever it costs.
Measured on a 1.4 GHz rv64, source and destination misaligned by one:
before after
memcmp 32K 34.4 458.0 MB/s
strncmp 32K 32.4 253.0 MB/s
strcmp 32K 41.0 280.0 MB/s
Each of those was the rate of the byte loop the word loop was meant to
replace. Pointers that genuinely disagree still take the byte loop, and
the aligned rates are unchanged.
The measurements come from the benchmark in apache/nuttx-apps#3706.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
The word loop walks src to a register boundary and then stores a whole
register at a time to dst, but nothing establishes that dst is on a
boundary too. Where the two pointers disagree about where a boundary
falls, every store in that loop is misaligned.
The base ISA does not require misaligned stores to be supported. Where
firmware emulates them each store traps into machine mode, and where
nothing emulates them the store faults, so this is not only a question of
speed. Measured on a 1.4 GHz rv64 that emulates them, with a 32 KB
string whose src and dst are misaligned by different amounts:
generic C 410.4 MB/s
this file 7.5 MB/s
which is around 178 cycles per byte, flat from 512 bytes to 32 KB.
Test the two pointers against each other before going wide, as
arch_strcpy.S already does. Pointers that agree still reach the word
loop, since walking src to a boundary walks dst to one as well; pointers
that disagree take the byte path, where no single boundary serves both.
After the change the misaligned case runs at 490 MB/s and the aligned
rates are unchanged.
The measurements come from the benchmark in apache/nuttx-apps#3706.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
Rewrite arch_memcpy.S and arch_memset.S to be register-width aware on
both RV32 and RV64 using REG_L/REG_S/SZREG macros from asm.h.
memcpy gains:
- 16xSZREG unrolled main loop (128B/iter on RV64, 64B on RV32).
- Shift-merge path for misaligned src: reads two aligned words
straddling each output word and shifts them together, so no load
or store is ever misaligned.
- Single SZREG and byte loops for remainder and small copies.
memset gains:
- 32xSZREG unrolled main loop (256B/iter on RV64, 128B on RV32)
using Duff's device for non-power-of-two remainders.
- .option norvc ensures fixed 4-byte instruction width for correct
jump offset calculation in the Duff's device entry.
- Zero-length input handled correctly (branch to guarded tail).
The old memcpy always used lw/sw even on RV64, wasting half the
memory bandwidth. The old memset unrolled only 16 bytes per iteration.
Signed-off-by: ganjing <ganjing@xiaomi.com>
Add word-at-a-time strlcpy using DETECTNULL for both the copy phase
and the strlen tail when truncated. The copy loop aligns src and
processes a register at a time, falling to bytewise for the last word
containing the terminator. When truncated, the remaining src length
is measured with a second word-at-a-time loop.
strlcpy has 46 call sites in a typical kernel image (more than strcpy)
and is not covered by newlib OPTSPEED, making it a high-value target.
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
Reduce branch overhead in the memcmp main loop by comparing four
words per iteration: XOR each pair, OR the four differences together,
and branch once. On a mismatch the single-word loop locates the
exact differing word within four words of the fault.
Add a beqz guard at .Lbyte_cmp entry to handle the case where the
4-word loop consumes all remaining bytes exactly.
Measured on QEMU RV32: memcmp(128) 313 -> 271 cycles (13% faster).
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
Add assembly-optimized implementations for 14 string/memory functions
using word-at-a-time techniques (DETECTNULL, broadcast+XOR) and
XLEN-adaptive macros for both RV32 and RV64:
- memmove: direction check + forward tail to memcpy, reverse path
with 16xSZREG unroll and shift-merge for misaligned src.
- memcmp: word-granularity compare when both pointers share alignment,
bytewise fallback for mismatched pointers.
- memchr: broadcast target byte, XOR with each word, DETECTNULL to
find matches. Counter-based bounds (no pointer overflow).
- strlen: DETECTNULL word loop, constants loaded from .srodata.
- strnlen: strlen with counter-based length limit.
- strcpy/strncpy: word loop with DETECTNULL, zero-fill remainder
for strncpy. strncpy reuses strcpy via #define USE_AS_STRNCPY.
- stpcpy/stpncpy: reuse strcpy/strncpy via #define USE_AS_STPCPY.
- strchr/strchrnul: broadcast+XOR detecting both target char and
null simultaneously. strchrnul reuses strchr via #define.
- strrchr: forward scan recording last match position.
- strncmp: word-at-a-time compare with null detection and counter.
- strcat: strlen(dst) then strcpy(dst_end, src) word-at-a-time.
Each function is independently selectable via CONFIG_RISCV_<FUNC>,
or all enabled together with CONFIG_RISCV_STRING_FUNCTION=y.
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ganjing <ganjing@xiaomi.com>
Implement architecture-specific ELF header definitions and relocation handling
for the MIPS architecture to enable loadable modules.
Fixes#19178.
Changes include:
- Add `arch/mips/include/elf.h` with MIPS ELF relocation types and
architecture-specific ELF data structures (`arch_elfdata_s`).
- Implement `libs/libc/machine/mips/arch_elf.c` containing `up_checkarch`,
`up_relocate`, and `up_relocateadd` functions handling `R_MIPS_NONE`,
`R_MIPS_32`, `R_MIPS_26`, `R_MIPS_HI16`, and `R_MIPS_LO16` relocations.
- Integrate MIPS machine-specific C library support in
`libs/libc/machine/mips/Make.defs`.
- Update `LDMODULEFLAGS` in `arch/mips/src/mips32/Toolchain.defs` to include the
little-endian (`-EL`) flag.
- Update `up_coherent_dcache` for proper cache synchronization on JZ4780.
Signed-off-by: Lwazi Dube <lwazeh@gmail.com>
1. The original method of restoring the register context relies on
the ret instruction, which does not conform to the semantics of
longjmp not returning.
2. There are csa leak during longjmp and need recycle.
Signed-off-by: zhangyu117 <zhangyu117@xiaomi.com>
Add setjmp/longjmp support for TriCore architecture using iLLD
intrinsics to save and restore the CSA (Context Save Area) chain.
This implementation saves upper/lower context registers and walks
the CSA linked list to restore the full call context on longjmp.
Signed-off-by: zhangyuan29 <zhangyuan29@xiaomi.com>
Add dedicated NEON implementations for mutually aligned medium and long memcpy copies when building with __ARM_NEON__. These paths use NEON multi-register loads and stores while preserving the existing VFP implementation for non-NEON VFP configurations.
NEON builds also define USE_VFP, so select the NEON implementation explicitly before falling back to VFP. Apply the same aligned-copy optimization to the armv7-a, armv7-r, and armv8-r implementations.
Signed-off-by: yaojiaqi <yaojiaqi@lixiang.com>
debug.h is a NuttX-specific, non-POSIX header. Placing it in the
top-level include/ directory creates naming conflicts with external
projects that define their own debug.h.
This commit moves the canonical header to include/nuttx/debug.h,
following the NuttX convention for non-POSIX/non-standard headers,
and updates all in-tree references.
A backward-compatibility shim is left at include/debug.h that
emits a deprecation #warning and re-includes <nuttx/debug.h>,
allowing out-of-tree code to continue building while migrating.
Signed-off-by: Piyush Patle <piyushpatle228@gmail.com>
Add support for ARMv6-M and ARMv8-M Baseline architectures
(Cortex-M0/M0+/M23) in the mcount profiling function. These cores
only support limited Thumb-1 instruction set and require different
assembly instructions compared to ARMv7-M and higher.
Changes:
- Use MOVS+BICS instead of BIC for bit clearing on M0/M23
- Separate register restore for limited push/pop instructions
- Use BX instead of direct POP to PC on M0/M23
Signed-off-by: yinshengkai <yinshengkai@bytedance.com>
when we build with greenhills compiler, if the exception_common function
is put into a separate section: ".text.exception_common", then the
address of exception_common function is an even number, for example "0x16a6c",
in this case, it will trigger the UsageFaults:INVSTATE error.
The reason for this error is that in the GHS compiler, a section declared
through the ".section" directive must specify at least the "a" attribute.
Otherwise, this section will not be linked into the final file, and this
is why the address of exception_common function is invalid.
The following is the official explanation in the GHS compiler documentation:
"The reason for this error is that in the GHS compiler, a section
declared through the .section directive must specify at least the "a"
attribute. Otherwise, this section will not be linked into the final
file. The following is the official explanation in the documentation:
"Sections that are intended be part of the final linked output should
have at least the `a` attribute"
and for GHS compiler, the .text section is recommended specify the
attribute of `ax`.
Signed-off-by: guoshichao <guoshichao@xiaomi.com>
This allows the compiler to automatically identify which string functions
can be compiled into libraries, and the compiler's internal implementation
is faster than libc functions.
Signed-off-by: anjiahao <anjiahao@xiaomi.com>
(gdb) bt
at /home/mi/ssd/dev-system/nuttx/include/string.h:321
ppcmd=<synthetic pointer>, vtbl=0x80069498) at nsh_parse.c:1909
param=0x800692b8) at nsh_parse.c:2593
cmdline=cmdline@entry=0x80069750 "hello") at nsh_parse.c:3028
argc=argc@entry=1, argv=argv@entry=0x80068870) at nsh_session.c:246
at nsh_consolemain.c:75
at nsh_main.c:74
...
Signed-off-by: yangao1 <yangao1@xiaomi.com>
riscv-none-elf-ld: /home/ligd/platform/dev-system/nuttx/staging/libc.a(arch_libc.o): in function `memcpy':
/home/ligd/platform/dev-system/nuttx/libs/libc/machine/arch_libc.c:131:(.text.memcpy+0x4e): undefined reference to `arch_memcpy'
Signed-off-by: ligd <liguiding1@xiaomi.com>
Add back these functions since clang will use them.
This reverts "libs/libc/arm: use builtin routines instead of aliases of __aeabi_mem*"
and adds source files to cmake.
Signed-off-by: chenxiaoyi <chenxiaoyi@xiaomi.com>
Fix an issue on ARMv8-R where NEON is not supported.
When NEON is disabled, the assembly-optimized implementations
of memset and memcpy should not be used.
Signed-off-by: Chengdong Wang <wangchengdong@lixiang.com>
Prior to this commit, it wasn't possible to load ELF modules from
the external PSRAM. There were two main issues about it: 1) copying
data using the instruction bus was being used instead of the data
bus (this, per si, isn't a problem, but requires special attention
regarding data alignment), and 2) the cache was not being properly
cleaned and flushed to properly access the loaded data using the
instruction bus.
Signed-off-by: Tiago Medicci Serrano <tiago.medicci@espressif.com>
Optimize crc32 standard(poly:0x04C11DB7) and crc32
castagnoli(poly:0x1EDC6F41) with arm crc32 extension instructions.
For example, crc32 standard caculates(lookup crc32 table) 1812 bytes data,
reduced the time from 118 us to 14 us through optimization.
Performance improved ~700%
Signed-off-by: Jinliang Li <lijinliang1@lixiang.com>
Summary:
- Replace direct use of `fence.i` instruction with `__ISB()` macro for instruction synchronization
- This change improves portability while maintaining the same functionality
Impact:
- No functional changes - both `fence.i` and `__ISB()` ensure instruction
synchronization on RISC-V
- Makes the code more maintainable by using the architecture abstraction
layer
Signed-off-by: Huang Qi <huangqi3@xiaomi.com>
It will be used to distinguish between hardware KASan and software KASan. Hardware KASan does not need to use plug-in
Signed-off-by: wangmingrong1 <wangmingrong1@xiaomi.com>
