We have implemented CI Build Retry for mitigating Download Failures. But right now it will reuse any Corrupted Downloads (instead of deleting them), which will cause CI Build Retry to fail repeatedly: https://github.com/apache/nuttx/actions/runs/24611381081/job/71966407518#step:10:627
```
Ignored files: system/argtable3/v3.2.0.7402e6e.tar.gz
gzip: stdin: not in gzip format
make[3]: *** [Makefile:76: argtable3] Error 2
```
This PR proposes to change `git clean -fd` to `git clean -xfdq`. This will remove any Corrupted Downloads, preventing their reuse for CI Build Retry:
```bash
## Simulate a corrupted download for argtable3
$ tools/configure.sh qemu-armv8a/netnsh
$ Downloading v3.2.0.7402e6e.tar.gz--argtable3
## Press Ctrl-C to stop the build
## `git clean -fd` won't remove the corrupted download
$ cd ../apps
$ git clean -fd
$ find . -name "v3.2.0.7402e6e.tar.gz"
./system/argtable3/v3.2.0.7402e6e.tar.gz
## `git clean -xfdq` removes the corrupted download
$ git clean -xfdq
$ find . -name "v3.2.0.7402e6e.tar.gz"
## Corrupted download is deleted correctly
```
`git clean -xfdq` is also found in the same testbuild script, so we are using it consistently:
12e8f92a28/tools/testbuild.sh (L286-L291)
Signed-off-by: Lup Yuen Lee <luppy@appkaki.com>
In Jan-Feb 2026: NuttX CI hit a [record high usage of GitHub Runners](https://github.com/apache/nuttx/issues/17914), exceeding the limit enforced by ASF Infrastructure Team. We analysed the PRs and discovered that most GitHub Runners were wasted on __(1) Failure to Download the Build Dependencies__ for DTC Device Tree, OpenAMP Messaging, MicroADB Debugger, MCUBoot Bootloader, NimBLE Bluetooth, etc __(2) Resubmitting PR Commits__:
- [Video: Analysing the Most Expensive PR](https://youtu.be/swFaxaTCEQg)
- [Video: Second Most Expensive PR](https://youtu.be/uSpQkzBogEw)
- [Video: Third Most Expensive PR](https://youtu.be/J7w1gyjwZ1w)
- [Video: Most Expensive Apps PR](https://youtu.be/182h8cRpfvI)
- [Spreadsheet: Most Expensive PRs](https://docs.google.com/spreadsheets/d/1HY7fIZzd_fs3QPyA0TX7vsYOjL86m1fNOf1Wls93luI/edit?gid=70515654#gid=70515654)
Why would __Download Failures__ waste GitHub Runners? That's because Download Failures will terminate the Entire CI Build (across All CI Jobs), requiring a restart of the CI Build. And the CI Build isn't terminated immediately upon failure: NuttX CI waits for the CI Job to complete (e.g. arm-01), before terminating the CI Build. Which means that CI Builds can get terminated 2.5 hours into the CI Build, wasting 2.5 elapsed hours x [7.4 parallel processes](https://lupyuen.org/articles/ci3#live-metric-for-full-time-runners) of GitHub Runners.
This PR proposes to __Retry the Build for Each CI Target__. NuttX CI shall rebuild each CI Target (e.g. `sim:nsh`), upon failure, up to 3 times (total 4 builds). Each rebuild will be attempted after a Randomised Delay with Exponential
Backoff, initially set to 60 seconds, then 120 seconds, 240 seconds. The rebuilds will mitigate the effects of Intermittent Download Failures that occur in GitHub Actions. (And eliminate developer frustration)
If the build fails after 3 retries: Subsequent CI Targets will __not be allowed to rebuild__ upon failure. This is to prevent cascading build failures from overloading GitHub Actions, and consuming too many GitHub Runners.
Note that NuttX CI shall retry the build for __Any Kind of Build Failure__, including Download Failures, Compile Errors and Config Errors. We designed it simplistically due to our current constraints: (1) Lack of CI Expertise (2) NuttX CI is Mission Critical (3) Legacy CI Scripts are Highly Complex. To prevent Compile Errors and Config Errors: We expect NuttX Devs to [Build and Test PRs in Our Own Repos](https://github.com/apache/nuttx/issues/18568), before submitting to NuttX.
What about __Resubmitting PR Commits__ and its wastage of GitHub Runners? We also require NuttX Devs to [Build and Test PRs in Our Own Repos](https://github.com/apache/nuttx/issues/18568), before resubmitting to NuttX. GitHub Runners will then be charged to the developer's quota, without affecting the GitHub Runners quota for Apache NuttX Project. We plan to [Kill All CI Jobs](https://youtu.be/182h8cRpfvI?si=MmAuwLISZPPMoqDq&t=1479) for PRs that have been switched to Draft Mode. We'll monitor this through the [NuttX Build Monitor](https://github.com/apache/nuttx/issues/18659).
Modified Files:
`tools/testbuild.sh`: We introduce a New Wrapper Function `retrytest` that will call the Existing Function `dotest`, to build the CI Target and retry on error.
`Documentation/components/tools/testbuild.rst`: Updated the `testbuild.sh` doc with the Retry Logic.
Signed-off-by: Lup Yuen Lee <luppy@appkaki.com>
Added break to the dotest Blacklist and CMake loop: it stops at the first match instead of continuing unnecessarily.
Signed-off-by: simbit18 <simbit18@gmail.com>
Executable files called "run" don't pass checkpath verification,
so we change the name to "run.sh"
Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
export currently build configuration and path to artifacts
so these can be used by NTFC run script
Signed-off-by: p-szafonimateusz <p-szafonimateusz@xiaomi.com>
Renamed savedefconfig -> refreshsilent name more consistent with the refresh stage for cmake on github.
Added new savedefconfig to fix
https://github.com/apache/nuttx/issues/14281
The simple improvement is designed to speed up compilation and reduce download errors on github and local.
Added a folder nxtmpdir for storing third-party packages
nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir
tools/Unix.mk:
added export NXTMPDIR := $(WSDIR)/nxtmpdir
tools/configure.sh:
added option -S creates the nxtmpdir folder for third-party packages.
tools/Config.mk:
added macro
CLONE - Git clone repository.
CHECK_COMMITSHA - Check if the branch contains the commit SHA-1.
tools/testbuild.sh:
added option -S
For now I added in the folder this package
ESP_HAL_3RDPARTY_URL = https://github.com/espressif/esp-hal-3rdparty.git
ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs
but you can also add other packages (maybe also of apps)
Most tools used for compliance and SBOM generation use SPDX identifiers
This change brings us a step closer to an easy SBOM generation.
Signed-off-by: Alin Jerpelea <alin.jerpelea@sony.com>
tools/testbuild.sh
added option -N Use CMake with Ninja as the backend.
updated help.
added date to print startup board
added HOST=Msys
github/workflows/build.yml
added option -N job docker
.github/workflows/build.yml: added CodeChecker support for GitHub Workflow
tools/testbuild.sh: added support for CodeChecker checks
- Added support for CodeChecker checks.
- Generate inspection reports and summaries.
- After the task is executed, the logs are compressed and the database is packaged.
Signed-off-by: xinbingnan <xinbingnan@xiaomi.com>
Summary:
- This commit adds exportandimport func for CONFIG_BUILD_KERNEL=y
Impact:
- ci with CONFIG_BUILD_KERNEL=y only
Testing:
- Tested with sabre-6quad:netknsh (will be updated later)
Signed-off-by: Masayuki Ishikawa <Masayuki.Ishikawa@jp.sony.com>
I plan to use this to run sim:ostest (and probably other tests later)
on the CI.
The script indirection might allow future non-sim usage as well.
(like running it with qemu, or even on the real hardware.)
I have no plan to do it by myself right now though.
An empty extension for sed -i is not widely available.
* Where it isn't available (eg. macOS's sed) "sed -i -e" will
create a backup file with the "-e" suffix.
* Even GNU sed documentation says it's "not recommended".
This commit deals with it by:
* Replace it with a more appropriate tool (kconfig-tweak)
* Or, specify the extension (.bak)
xargs for macOS does not support the '-a/--file-arg' flag so
build artifacts were not getting stored. This change passes it in
via stdin which is more portable.
Signed-off-by: Brennan Ashton <bashton@brennanashton.com>
Add new option -A is added to tools/testbuild.sh that will take the
created build executable and store it in a folder for the config
that generated it under $ARTIFACTDIR which can be set via an
environment variable or defaulted to $(TOPDIR)/buildartifacts
This is also helpful for local testing because you can now run
tools/testbuild.sh -A sim.dat and have all of the simulation
targets generated without having to rebuild along the way.
In the GitHub Actions workflow the artifacs are uploaded
two two bundles one for macOS and one for Linux
Signed-off-by: Brennan Ashton <bashton@brennanashton.com>
since it doesn't need anymore with the follow patch:
commit e7d9260014
Author: Xiang Xiao <xiaoxiang@xiaomi.com>
Date: Mon Feb 17 20:19:25 2020 +0800
arch: Customize the typedef of size_t instead of intptr_t
To ensure size_t same as toolchain definition in the first place and
rename CXX_NEWLONG to ARCH_SIZET_LONG. The change also check whether
__SIZE_TYPE__ exist before CONFIG_ARCH_SIZET_LONG so our definition
can align with toolchain(gcc/clang) definition automatically.
Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>
Since testbuild.sh called by nightly build and check build, it should
suppress logs from configure stage to keep all output clean.
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
Also call configure.sh with JOPTION to enable parallel build
Change-Id: I32c4e77fb30c40d8d424159cc0871b8c3e3f10b6
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
As in https://github.com/apache/incubator-nuttx/issues/1114 discuss, let
EXTRA_FLAGS="EXTRAFLAGS=" defaultly to avoid testbuild.sh called without
-e option fail issue. It also works well with -e "-Wno-cpp -Werror".
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
Aim to let makefunc caller just pass in the target. So update makefunc to hardcode make/extra
flags options and redirection, meanwhile avoid the case in which makefunc would remove the quotes
for EXTRAFLAGS. Note that JOPTIONS should also hardcode once parallel build break finally fixed.
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
Call 'tools/testbuild.sh -e -Wno-cpp testlist' would pass "EXTRAFLAGS=-Wno-cpp"
to make command line. Then it could suppress the warnings with pre-processor
directive #warning in GCC.
Change-Id: I61e5f9b3774149f64bdd625677cc9aabaa2fea90
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
Redirect configure.sh stdout to /dev/null to suppress lots of log
since configure.sh behaviour updated.
Signed-off-by: liuhaitao <liuhaitao@xiaomi.com>
