mirror of
https://github.com/apache/nuttx.git
synced 2026-10-03 04:07:55 +00:00
44cdacf607
The memory report builds this pull request merged into master, together with the nuttx-apps default branch, and nothing else, so a companion or predecessor pull request it declares is absent. For a breaking change the target then fails to build and no report is produced at all, even though the Build workflow already tests the declared sources through Depends-On. Apply the declared dependencies before building, reusing the parser and the fetch/cherry-pick sequence that build.yml uses, and mapping each repository to its checkout exactly as build.yml does. A stacked nuttx dependency is no more optional than an apps one: a pull request that uses an API its predecessor introduces does not build without it. As build.yml does, read the description through the API rather than trusting the event payload, so that a manual re-run after editing a Depends-On line applies the current declaration instead of the one the run was created with. Unlike build.yml, a failed read stops the job rather than falling back to the payload: build.yml resolves this once and hands every target the same tree, while this job runs per target, so a fallback could leave targets on different declarations while their results are filed under one SHA. A declared dependency that cannot be applied fails the job, and so does a missing parser, a parser crash, or a status this step does not recognise: each of those means the declaration was never evaluated, and continuing would measure a combination nobody asked for. build.yml fails Fetch-Source on the same conditions, and no other step in this job carries continue-on-error, so falling back silently would be inconsistent with both. A declaration that parses to nothing valid only warns, again matching build.yml. Forward the parser's warnings too. --print-state prints only the state, so an entry the parser drops -- an unsupported repository, say -- would otherwise leave no trace here at all, although build.yml annotates it, and the source set named below would be silently incomplete. The report is filed under the pull request head SHA rather than the SHA of the tree that was built, so the measurement cannot be reproduced from that SHA alone and cannot be split per dependency. That limits provenance, not the measurement: a combined result is what the declaration asks for, and a regression that only appears in combination is still a regression. Name the whole source set in the step summary so the reader knows which heads went into the number. Note in the parser that the --print-state output is a parsed contract; the edit gate that used to be its only caller is gone. Update the CI documentation to match. Its Pull Request Dependencies section attributes dependency application to build.yml's Fetch-Source job alone, so after this change it would read as if the memory report measured the normal source selection. Cross-reference the two sections rather than restating the rules, which stay shared. Signed-off-by: zhangning21 <zhangning21@xiaomi.com>
434 lines
18 KiB
YAML
434 lines
18 KiB
YAML
name: MemBrowse Memory Report
|
|
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches:
|
|
- master
|
|
- "releases/*"
|
|
|
|
# pull-requests read: needed to re-read edited Depends-On declarations on re-runs.
|
|
permissions:
|
|
contents: read
|
|
pull-requests: read
|
|
|
|
# Per-PR group so superseded PR pushes cancel; per-SHA on push so master
|
|
# commits never share a group. A shared refs/heads/master group lets a burst
|
|
# of pushes evict each other while pending (GitHub keeps only one pending run
|
|
# per group), leaving a commit with no report and breaking the MemBrowse
|
|
# parent chain for every commit that bases on it.
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
jobs:
|
|
# Detect whether any source files (i.e. non-doc/CODEOWNERS) changed.
|
|
# When only docs/CODEOWNERS change we skip the build and post an "identical"
|
|
# MemBrowse report instead.
|
|
changes-filter:
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
source: ${{ steps.filter.outputs.source }}
|
|
steps:
|
|
# Shallow checkout; the base commit needed for the diff is fetched
|
|
# explicitly below instead of cloning the repo's full history.
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
fetch-depth: 2
|
|
# Detect whether any non-doc/CODEOWNERS source files changed.
|
|
# Implemented with plain git instead of a third-party paths-filter
|
|
# action, since Apache's GitHub Actions allowlist forbids actions that
|
|
# aren't GitHub-created or explicitly permitted.
|
|
- id: filter
|
|
env:
|
|
BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
|
|
HEAD_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
|
|
GH_TOKEN: ${{ github.token }}
|
|
run: |
|
|
# On the very first push to a branch, github.event.before is all-zeros.
|
|
if [ -z "$BASE_SHA" ] || [ "$BASE_SHA" = "0000000000000000000000000000000000000000" ]; then
|
|
echo "source=true" >> "$GITHUB_OUTPUT"
|
|
echo "No usable base SHA; treating as source change."
|
|
exit 0
|
|
fi
|
|
# Diff from the merge-base (fork point) so only this PR/push's own
|
|
# commits count. For pull_request events BASE_SHA is the current base
|
|
# branch tip, which drifts ahead of the fork point as the base branch
|
|
# advances; diffing from the merge-base isolates the PR's real changes
|
|
# (e.g. keeps a docs-only PR classified as docs-only).
|
|
#
|
|
# GitHub's compare API uses 3-dot semantics, so .merge_base_commit.sha
|
|
# is the fork point. gh is pre-installed on the runner.
|
|
MERGE_BASE=$(gh api \
|
|
"repos/$GITHUB_REPOSITORY/compare/$BASE_SHA...$HEAD_SHA" \
|
|
--jq '.merge_base_commit.sha' 2>/dev/null || true)
|
|
if [ -z "$MERGE_BASE" ]; then
|
|
# API unavailable/unexpected: fail safe toward building.
|
|
echo "source=true" >> "$GITHUB_OUTPUT"
|
|
echo "Could not resolve merge-base via API; treating as source change."
|
|
exit 0
|
|
fi
|
|
echo "Merge-base: $MERGE_BASE"
|
|
# The shallow checkout has HEAD but may lack the merge-base commit;
|
|
# fetch just that one commit so the local diff can read its tree.
|
|
git cat-file -e "$MERGE_BASE^{commit}" 2>/dev/null \
|
|
|| git fetch --depth=1 origin "$MERGE_BASE" 2>/dev/null || true
|
|
# MERGE_BASE is an ancestor of HEAD, so this diff yields the PR/push's
|
|
# own changes.
|
|
if ! changed=$(git diff --name-only "$MERGE_BASE" "$HEAD_SHA" 2>/dev/null); then
|
|
echo "source=true" >> "$GITHUB_OUTPUT"
|
|
echo "git diff against merge-base failed; treating as source change."
|
|
exit 0
|
|
fi
|
|
echo "Changed files:"
|
|
echo "$changed"
|
|
# Drop paths that should NOT count as source changes; if anything
|
|
# survives, real source changed.
|
|
source=$(echo "$changed" | grep -vE \
|
|
-e '^AUTHORS$' \
|
|
-e '^CONTRIBUTING\.md$' \
|
|
-e '(^|/)CODEOWNERS$' \
|
|
-e '^Documentation/' \
|
|
-e '^tools/ci/docker/linux/' \
|
|
-e '^tools/codeowners/[^/]+$' \
|
|
|| true)
|
|
if [ -n "$source" ]; then
|
|
echo "source=true" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "source=false" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
|
|
load-targets:
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
matrix: ${{ steps.set-matrix.outputs.matrix }}
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
- id: set-matrix
|
|
run: echo "matrix=$(jq -c '.' .github/membrowse-targets.json)" >> $GITHUB_OUTPUT
|
|
|
|
# Post an "identical" MemBrowse report when only docs/CODEOWNERS changed.
|
|
# Only runs on push events (master/releases/tags) to keep the baseline
|
|
# complete; PR events don't produce identical markers.
|
|
identical:
|
|
needs: [changes-filter, load-targets]
|
|
if: needs.changes-filter.outputs.source == 'false' && github.event_name == 'push'
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include: ${{ fromJson(needs.load-targets.outputs.matrix) }}
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
with:
|
|
fetch-depth: 2
|
|
- uses: actions/setup-python@v7
|
|
with:
|
|
python-version: '3.11'
|
|
- name: Install membrowse
|
|
run: pip install --no-cache-dir membrowse
|
|
- name: Upload identical - ${{ matrix.target_name }}
|
|
env:
|
|
MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }}
|
|
MEMBROWSE_API_URL: ${{ vars.MEMBROWSE_API_URL }}
|
|
TARGET_NAME: ${{ matrix.target_name }}
|
|
# Base on the previous branch tip, not the (often unreported mid-PR)
|
|
# git parent.
|
|
BEFORE_SHA: ${{ github.event.before }}
|
|
run: |
|
|
ARGS=(--upload --github
|
|
--target-name "$TARGET_NAME"
|
|
--api-key "$MEMBROWSE_API_KEY"
|
|
--identical)
|
|
if [ -n "$MEMBROWSE_API_URL" ]; then
|
|
ARGS+=(--api-url "$MEMBROWSE_API_URL")
|
|
fi
|
|
# Override parent with previous tip; skip on branch creation (0s).
|
|
if [ -n "$BEFORE_SHA" ] && [ "$BEFORE_SHA" != "0000000000000000000000000000000000000000" ]; then
|
|
ARGS+=(--base-sha "$BEFORE_SHA")
|
|
fi
|
|
membrowse report "${ARGS[@]}"
|
|
|
|
# Build target with debug symbols + linker map, then upload MemBrowse report.
|
|
# Uses the NuttX CI Docker image so toolchains, kconfig-frontends, etc. are
|
|
# already installed.
|
|
analyze:
|
|
needs: [changes-filter, load-targets]
|
|
if: needs.changes-filter.outputs.source == 'true'
|
|
runs-on: ubuntu-latest
|
|
env:
|
|
DOCKER_BUILDKIT: 1
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include: ${{ fromJson(needs.load-targets.outputs.matrix) }}
|
|
steps:
|
|
- name: Checkout nuttx repo
|
|
uses: actions/checkout@v7
|
|
with:
|
|
path: sources/nuttx
|
|
fetch-depth: 2
|
|
|
|
- name: Checkout nuttx-apps repo
|
|
uses: actions/checkout@v7
|
|
with:
|
|
repository: apache/nuttx-apps
|
|
path: sources/apps
|
|
fetch-depth: 1
|
|
|
|
# Free space before pulling the large NuttX CI image; ubuntu-latest ships
|
|
# only ~14 GB free on / and the image otherwise fails to unpack with
|
|
# "no space left on device". Mirrors the cleanup in .github/workflows/build.yml.
|
|
- name: Show Disk Space
|
|
run: df -h
|
|
|
|
- name: Free Disk Space (Ubuntu)
|
|
run: |
|
|
sudo rm -rf /usr/local/lib/android
|
|
|
|
- name: After CLEAN-UP Disk Space
|
|
run: df -h
|
|
|
|
# Apply declared nuttx/apps dependencies before measuring, matching
|
|
# build.yml's checkout mapping and cherry-pick order. API or parser
|
|
# failures and parsed dependencies that cannot be applied are fatal, so a
|
|
# report is never taken from the wrong source set. Keep this after the
|
|
# disk cleanup: applying a dependency deepens a checkout.
|
|
- name: Apply depends-on PRs
|
|
if: ${{ github.event_name == 'pull_request' && github.base_ref == 'master' }}
|
|
shell: bash
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
PR_BODY: ${{ github.event.pull_request.body }}
|
|
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
run: |
|
|
set -uo pipefail
|
|
|
|
# Re-read the current body so a re-run picks up an edited Depends-On
|
|
# line. Do not fall back to the event payload: unlike build.yml, which
|
|
# resolves this once for every target, each matrix leg resolves
|
|
# independently and could otherwise use a different declaration.
|
|
if ! PR_BODY="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" --jq '.body // ""')"; then
|
|
echo "::error::Could not read the PR description."
|
|
exit 1
|
|
fi
|
|
|
|
# A missing parser, a crash, or an unknown status all leave the
|
|
# declaration unevaluated.
|
|
PARSER=sources/nuttx/.github/scripts/depends_on.py
|
|
if [ ! -f "$PARSER" ]; then
|
|
echo "::error::${PARSER} not found."
|
|
exit 1
|
|
fi
|
|
|
|
STATE="${RUNNER_TEMP:-/tmp}/depends-on-state.txt"
|
|
if ! python3 "$PARSER" --print-state > "$STATE"; then
|
|
echo "::error::Could not parse the depends-on declarations."
|
|
exit 1
|
|
fi
|
|
|
|
# --print-state omits warnings. Run --github-output again with its
|
|
# outputs and report discarded so dropped entries remain visible as
|
|
# workflow annotations.
|
|
if ! WARNINGS="$(GITHUB_OUTPUT=/dev/null REPORT_PATH= python3 "$PARSER" --github-output)"; then
|
|
echo "::error::Could not re-read the depends-on warnings."
|
|
exit 1
|
|
fi
|
|
printf '%s\n' "$WARNINGS" | grep '^::warning::' || true
|
|
|
|
STATUS=$(head -n 1 "$STATE")
|
|
case "$STATUS" in
|
|
ok) ;;
|
|
none)
|
|
echo "No Depends-On declaration; building against the default sources."
|
|
exit 0 ;;
|
|
invalid)
|
|
# The parser warning forwarded above already explains this.
|
|
echo "No valid dependency parsed; building against the default sources."
|
|
exit 0 ;;
|
|
*)
|
|
echo "::error::Unexpected depends-on parser status: ${STATUS}"
|
|
exit 1 ;;
|
|
esac
|
|
|
|
git config --global user.email "actions@github.com"
|
|
git config --global user.name "github-actions"
|
|
|
|
# ok must carry at least one reference; an unreadable or empty list
|
|
# would skip the loop and report the default sources as though the
|
|
# declaration had been applied. Keep the loop in this shell, not a
|
|
# pipeline, so a dependency failure exits the step.
|
|
if ! tail -n +2 "$STATE" > "${STATE}.refs" || [ ! -s "${STATE}.refs" ]; then
|
|
echo "::error::Could not read the parsed dependency list."
|
|
exit 1
|
|
fi
|
|
# The report is keyed by the pull request head SHA, which names
|
|
# neither checkout: nuttx is the pull request merged into its base,
|
|
# and apps is an unpinned default branch the event never mentions.
|
|
# Record both in the summary, before anything is applied, so the
|
|
# source set is visible where the report is read.
|
|
NUTTX_CHECKOUT_SHA=$(git -C sources/nuttx rev-parse HEAD)
|
|
APPS_CHECKOUT_SHA=$(git -C sources/apps rev-parse HEAD)
|
|
{
|
|
echo "### Depends-On source set"
|
|
echo
|
|
echo "Measured under this pull request's head SHA, together with:"
|
|
echo "- \`apache/nuttx\` checkout @ \`${NUTTX_CHECKOUT_SHA}\`"
|
|
echo "- \`apache/nuttx-apps\` checkout @ \`${APPS_CHECKOUT_SHA}\`"
|
|
} >> "${GITHUB_STEP_SUMMARY:-/dev/null}"
|
|
|
|
while read -r DEP; do
|
|
[ -n "$DEP" ] || continue
|
|
DEP_REPO=${DEP%/pull/*}
|
|
DEP_NUM=${DEP##*/}
|
|
|
|
# Keep the repository mapping aligned with build.yml.
|
|
case "$DEP_REPO" in
|
|
"apache/nuttx") REPO_PATH=sources/nuttx ;;
|
|
"apache/nuttx-apps") REPO_PATH=sources/apps ;;
|
|
*)
|
|
echo "::error::Unsupported dependency repository: ${DEP_REPO}"
|
|
exit 1 ;;
|
|
esac
|
|
|
|
echo "Applying dependency ${DEP}"
|
|
# Deepening is best effort; the common-base and rev-list checks
|
|
# below still fail closed. Say it happened, so a missing common
|
|
# base is not mistaken for an unrelated dependency.
|
|
if [ -f "${REPO_PATH}/.git/shallow" ] \
|
|
&& ! git -C "$REPO_PATH" fetch --unshallow origin; then
|
|
echo "::warning::Could not deepen ${REPO_PATH}; a missing common base below may follow from that rather than from ${DEP}."
|
|
fi
|
|
if ! git -C "$REPO_PATH" fetch origin "pull/${DEP_NUM}/head:dep-${DEP_NUM}"; then
|
|
echo "::error::Could not fetch ${DEP} (the PR may not exist)."
|
|
exit 1
|
|
fi
|
|
|
|
# Reject unrelated histories before computing HEAD..dep, which
|
|
# would otherwise list every dependency commit.
|
|
if [ -z "$(git -C "$REPO_PATH" merge-base "dep-${DEP_NUM}" HEAD || true)" ]; then
|
|
echo "::error::Could not find a common base with ${DEP}."
|
|
exit 1
|
|
fi
|
|
if ! COMMITS=$(git -C "$REPO_PATH" rev-list --reverse "HEAD..dep-${DEP_NUM}"); then
|
|
echo "::error::Could not list the commits of ${DEP}."
|
|
exit 1
|
|
fi
|
|
|
|
DEP_SHA=$(git -C "$REPO_PATH" rev-parse "dep-${DEP_NUM}")
|
|
if [ -z "$COMMITS" ]; then
|
|
echo "Dependency ${DEP} is already included."
|
|
echo "- \`${DEP}\` @ \`${DEP_SHA}\` (already in the checkout)" \
|
|
>> "${GITHUB_STEP_SUMMARY:-/dev/null}"
|
|
continue
|
|
fi
|
|
|
|
# shellcheck disable=SC2086
|
|
if ! git -C "$REPO_PATH" cherry-pick $COMMITS; then
|
|
echo "::error::Could not cherry-pick ${DEP}."
|
|
echo "::error::If your pull request contains merge commits, rebase instead of merging."
|
|
git -C "$REPO_PATH" cherry-pick --abort || true
|
|
exit 1
|
|
fi
|
|
echo "Applied ${DEP} @ ${DEP_SHA}"
|
|
echo "- \`${DEP}\` @ \`${DEP_SHA}\`" \
|
|
>> "${GITHUB_STEP_SUMMARY:-/dev/null}"
|
|
done < "${STATE}.refs"
|
|
|
|
- name: Docker Login
|
|
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.actor }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Docker Pull
|
|
run: docker pull ghcr.io/apache/nuttx/apache-nuttx-ci-linux
|
|
|
|
- name: Build ${{ matrix.target_name }}
|
|
uses: ./sources/nuttx/.github/actions/ci-container
|
|
with:
|
|
run: |
|
|
git config --global --add safe.directory /github/workspace/sources/nuttx
|
|
git config --global --add safe.directory /github/workspace/sources/apps
|
|
cd /github/workspace/sources/nuttx
|
|
./tools/configure.sh -l ${{ matrix.board_config }}
|
|
echo CONFIG_DEBUG_SYMBOLS=y >> .config
|
|
echo CONFIG_DEBUG_LINK_MAP=y >> .config
|
|
if [ -n "${{ matrix.config_overrides }}" ]; then
|
|
kconfig-tweak ${{ matrix.config_overrides }}
|
|
fi
|
|
make olddefconfig
|
|
# Preserve preprocessed linker scripts (.ld.tmp) so MemBrowse can
|
|
# read them. Arch Makefiles delete these immediately after linking.
|
|
find arch -name Makefile -exec sed -i '/DELFILE, $(addsuffix .tmp,$(ARCHSCRIPT))/d' {} +
|
|
make -j$(nproc)
|
|
|
|
# MemBrowse action runs from $GITHUB_WORKSPACE and discovers git via CWD,
|
|
# but nuttx is checked out to sources/nuttx/ (so apps can sit beside it).
|
|
# Expose the repo's .git at the workspace root so git metadata resolves.
|
|
- name: Expose nuttx .git to workspace root
|
|
run: ln -sfn sources/nuttx/.git .git
|
|
|
|
- name: Prefix linker script paths
|
|
id: ld
|
|
run: |
|
|
prefixed=""
|
|
for p in ${{ matrix.ld }}; do
|
|
prefixed="$prefixed sources/nuttx/$p"
|
|
done
|
|
paths=$(echo $prefixed | xargs)
|
|
echo "paths=$paths" >> "$GITHUB_OUTPUT"
|
|
echo "Resolved ld paths: [$paths]"
|
|
for p in $paths; do
|
|
if [ -e "$p" ]; then
|
|
echo " OK $p ($(stat -c '%s bytes, owner=%U:%G' "$p" 2>/dev/null))"
|
|
else
|
|
echo " MISS $p"
|
|
echo " ls dir:"
|
|
ls -la "$(dirname "$p")" 2>&1 | head -30
|
|
fi
|
|
done
|
|
|
|
- uses: actions/setup-python@v7
|
|
with:
|
|
python-version: '3.11'
|
|
- name: Install membrowse
|
|
run: pip install --no-cache-dir membrowse
|
|
- name: MemBrowse analysis - ${{ matrix.target_name }}
|
|
env:
|
|
MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }}
|
|
MEMBROWSE_API_URL: ${{ vars.MEMBROWSE_API_URL }}
|
|
TARGET_NAME: ${{ matrix.target_name }}
|
|
ELF: sources/nuttx/${{ matrix.elf }}
|
|
LD_PATHS: ${{ steps.ld.outputs.paths }}
|
|
MAP_FILE: ${{ matrix.map_file != '' && format('sources/nuttx/{0}', matrix.map_file) || '' }}
|
|
LINKER_VARS: ${{ matrix.linker_vars }}
|
|
# Push only: base on the previous branch tip, not the (often
|
|
# unreported mid-PR) git parent. Empty on PRs (--github bases on the
|
|
# PR target tip).
|
|
BEFORE_SHA: ${{ github.event_name == 'push' && github.event.before || '' }}
|
|
run: |
|
|
set -o pipefail
|
|
ARGS=("$ELF" "$LD_PATHS"
|
|
--upload --github
|
|
--target-name "$TARGET_NAME"
|
|
--api-key "$MEMBROWSE_API_KEY")
|
|
if [ -n "$MEMBROWSE_API_URL" ]; then
|
|
ARGS+=(--api-url "$MEMBROWSE_API_URL")
|
|
fi
|
|
# Override parent with previous tip; skip on branch creation (0s).
|
|
if [ -n "$BEFORE_SHA" ] && [ "$BEFORE_SHA" != "0000000000000000000000000000000000000000" ]; then
|
|
ARGS+=(--base-sha "$BEFORE_SHA")
|
|
fi
|
|
if [ -n "$MAP_FILE" ]; then
|
|
ARGS+=(--map-file "$MAP_FILE")
|
|
fi
|
|
if [ -n "$LINKER_VARS" ]; then
|
|
set -f
|
|
for var in $LINKER_VARS; do
|
|
ARGS+=(--def "$var")
|
|
done
|
|
set +f
|
|
fi
|
|
membrowse --verbose INFO report "${ARGS[@]}"
|