Skip to content

feat(extensions): sysrq-serial-trigger + kernel-debug-tiers (on-device kernel debugging via serial console)#9776

Open
iav wants to merge 3 commits intomainfrom
feat/sysrq-serial-trigger
Open

feat(extensions): sysrq-serial-trigger + kernel-debug-tiers (on-device kernel debugging via serial console)#9776
iav wants to merge 3 commits intomainfrom
feat/sysrq-serial-trigger

Conversation

@iav
Copy link
Copy Markdown
Contributor

@iav iav commented May 7, 2026

Description

Two extensions that, together with the kernel patches already merged as PR #9750 (rk3399 8250_dw) and PR #9760 (mvebu-6.18 8250_dw), form a coherent system for on-device kernel debugging through the serial console on headless ARM boards (NAS-style: helios4, helios64, similar).

Each piece is necessary, none is sufficient on its own:

  • kernel-debug-tiers turns on the kernel-side information without which there is nothing to debug — BTF for hung-task tracebacks, printk timestamps and PRINTK_CALLER, lockup/hung-task detection, pstore/ramoops for post-reboot dmesg, KGDB symbol resolution. Three cumulative tiers (KERNEL_DEBUG_TIER=1|2|3, default 1) so the operator picks the cost they're willing to pay; tier 0 is a deliberate no-op.

  • sysrq-serial-trigger turns on the serial console as a debug surface — MAGIC_SYSRQ + MAGIC_SYSRQ_SERIAL_SEQUENCE in kconfig, kernel.sysrq=1 sysctl, and a long enough u-boot BOOTDELAY for the operator to actually catch the prompt. Without this, even a kernel full of debug info can only be examined post-mortem; nothing can be done about a live hang.

  • The 8250_dw driver patches (PR fix(kernel/rockchip64-7.0): restore SysRq-via-BREAK on dw-apb-uart (8250_dw) #9750/fix(kernel/mvebu-6.18): restore SysRq-via-BREAK on dw-apb-uart (8250_dw) #9760, already merged) close the last gap on rk3399 and mvebu: the BREAK condition on a dw-apb-uart goes through dw8250_handle_irq(), which used to swallow LSR.BI without invoking uart_handle_break()handle_sysrq_from_serial(). Before those patches, even a fully configured SysRq-over-serial setup on these SoCs only appeared to work; the operator's BREAK never reached the SysRq dispatcher.

Together these three changes make the well-documented "send BREAK, type the magic sequence, then press a SysRq command" workflow actually functional on rk3399 and mvebu boards, with enough kernel debug info behind it that the resulting tracebacks and KGDB sessions are informative rather than just hex addresses.

Tier guarantees

  • Hard-fail on conflicting BTF preference: KERNEL_BTF=no + KERNEL_DEBUG_TIER>=1 exits early in extension_prepare_config with a helpful message (set KERNEL_BTF=yes/leave unset, or KERNEL_DEBUG_TIER=0).
  • Tier 0 is a real opt-out (extension stays loaded, kernel side off) — useful in shared configs where some boards must keep KERNEL_BTF=no for RAM reasons.
  • sysrq-serial-trigger is independent of the debug tiers; either can be enabled standalone.

Tests performed

End-to-end on helios64 (rockchip64-edge, kernel 7.0.3-rockchip64):

Run Override Expected Result
T1 KERNEL_DEBUG_TIER=1 (default) full kernel build, tier 1 markers + sysrq markers ✓ build OK, kernel 7.0.3-rockchip64, 40m30s
T0 KERNEL_DEBUG_TIER=0 extension loaded, no tier-N kconfig changes
T2 KERNEL_DEBUG_TIER=2 tier 1 + tier 2 (pstore), no tier 3
T3 KERNEL_DEBUG_TIER=3 tier 1 + tier 2 + tier 3 (KGDB)
BTF guard KERNEL_BTF=no (default tier 1) hard-fail in extension_prepare_config ✓ rc=43, exact error message
BTF=no + T0 KERNEL_BTF=no KERNEL_DEBUG_TIER=0 guard skipped, build proceeds

Checklist

Assisted-by: Claude:claude-opus-4.7

Summary by CodeRabbit

  • New Features
    • Multi-tier kernel debugging (tiers 0–3) providing incremental diagnostics: no-op tier, hung-task/stack checks, pstore/ramoops support, and optional KGDB/KDB over serial with required boot args.
    • Magic SysRq over serial plus autoboot adjustment and a sysctl to enable sysrq on target images, allowing serial operators to interrupt and control boot behavior.

iav added 2 commits May 7, 2026 06:05
Enable Magic SysRq through the serial console for headless ARM boards
(Helios64, etc.). Mainline ships MAGIC_SYSRQ_SERIAL_SEQUENCE empty,
disabling BREAK-triggered SysRq; this extension fills it with a
deliberate sequence after BREAK so the operator on the serial console
can sync/remount-RO/reboot from any kernel state where interrupts flow.
Also overrides distro kernel.sysrq=176 to 1 (full command set) and
extends u-boot autoboot delay so the prompt is actually catchable.

Pair with the companion `kernel-debug-tiers` extension to make the
tracebacks reachable via SysRq actually informative (BTF, hung-task,
KGDB symbol resolution).

Signed-off-by: Igor Velkov <325961+iav@users.noreply.github.com>
Cumulative kernel debug-information tiers for headless boards that need
on-device debugging through the serial console:

  KERNEL_DEBUG_TIER=0    no-op (extension loaded but kernel-side disabled)
  KERNEL_DEBUG_TIER=1    printk timestamps + lockup/hung-task detection
                         (default — cheap, no board prerequisites)
  KERNEL_DEBUG_TIER=2    + pstore/ramoops (needs DT or bootarg reservation)
  KERNEL_DEBUG_TIER=3    + KGDB/KDB over serial (needs kgdboc= bootarg)

By itself the extension does nothing visible at runtime; it bakes enough
information into the kernel image that operator-console facilities like
SysRq tracebacks and KGDB sessions are actually useful instead of streams
of hex addresses. Pair with `sysrq-serial-trigger` (separate extension)
to also enable the operator-control surface itself.

extension_prepare_config validates the tier value, hard-fails with a
helpful message on KERNEL_BTF=no + KERNEL_DEBUG_TIER>=1 (BTF is required
for hung-task tracebacks and KGDB symbol resolution; either set
KERNEL_BTF=yes or KERNEL_DEBUG_TIER=0 to opt out of the kernel side),
and forces KERNEL_BTF=yes for tier>=1 when not explicitly set.

Signed-off-by: Igor Velkov <325961+iav@users.noreply.github.com>
@iav iav requested a review from igorpecovnik as a code owner May 7, 2026 03:26
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 7, 2026

Review Change Stack
No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4229acef-4a73-41c8-b63c-b7ba91b38b41

📥 Commits

Reviewing files that changed from the base of the PR and between 30c1112 and 205685c.

📒 Files selected for processing (1)
  • extensions/sysrq-serial-trigger.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • extensions/sysrq-serial-trigger.sh

📝 Walkthrough

Walkthrough

Two new Bash extensions: kernel-debug-tiers adds tiered kernel debug validation and tiered kernel config (tiers 0–2 in these diffs), and sysrq-serial-trigger enables Magic SysRq over serial with U-Boot autoboot tweaks and an image sysctl to enable SysRq.

Changes

Kernel Debug Tiers

Layer / File(s) Summary
Parameter Validation
extensions/kernel-debug-tiers.sh
Validates KERNEL_DEBUG_TIER is 0–3; tier 0 treated as kernel-side no-op with early return; tiers ≥1 enforce KERNEL_BTF=yes or fail and emit alerts.
Tier 1 Kernel Config
extensions/kernel-debug-tiers.sh
Applies printk timestamps/callers, hung-task and soft-lockup detection, stack-end checks, and sets DEFAULT_HUNG_TASK_TIMEOUT=120 when tier ≥1.
Tier 2 Pstore/Ramoops
extensions/kernel-debug-tiers.sh
Enables PSTORE/ramoops-related kernel options (console, RAM, deflate) when tier ≥2.

SysRq Serial Trigger

Layer / File(s) Summary
Kernel Configuration
extensions/sysrq-serial-trigger.sh
Enables MAGIC_SYSRQ and MAGIC_SYSRQ_SERIAL; forces MAGIC_SYSRQ_DEFAULT_ENABLE=1; validates non-empty SYSRQ_SERIAL_SEQUENCE and sets MAGIC_SYSRQ_SERIAL_SEQUENCE either in .config or via hash-only modification list.
U-Boot Configuration
extensions/sysrq-serial-trigger.sh
Sets CONFIG_BOOTDELAY=5 and enables CONFIG_AUTOBOOT_NEVER_TIMEOUT using U-Boot's scripts/config.
Userland Configuration
extensions/sysrq-serial-trigger.sh
Writes /etc/sysctl.d/60-armbian-sysrq.conf with kernel.sysrq = 1 into the image to override distro sysctl defaults.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 In config fields where kernels dream,
Tiers and sysrq stitch the seam,
Serial whispers, bootdelay tamed—
A rabbit hops, the logs named.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: two new extensions (sysrq-serial-trigger and kernel-debug-tiers) for on-device kernel debugging via serial console, matching the file additions and objectives.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/sysrq-serial-trigger

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added size/medium PR with more then 50 and less then 250 lines 05 Milestone: Second quarter release Needs review Seeking for review Framework Framework components labels May 7, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@extensions/sysrq-serial-trigger.sh`:
- Around line 48-65: The code accepts an empty SYSRQ_SERIAL_SEQUENCE which, when
set to "", will silently disable serial SysRq; change the logic around seq
(declared from SYSRQ_SERIAL_SEQUENCE) to guard against empty values: if seq is
empty or only whitespace, treat it as unset and skip calling
kernel_config_set_string and avoid appending to kernel_config_modifying_hashes
(or set a safe default sequence), and ensure opts_y/opts_val still reflect
enabling only when a non-empty sequence is provided; update the conditional that
writes MAGIC_SYSRQ_SERIAL_SEQUENCE so kernel_config_set_string and
kernel_config_modifying_hashes are only invoked when seq is non-empty (use the
existing seq variable and the functions kernel_config_set_string and
kernel_config_modifying_hashes).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9620094e-8b87-4446-aa9a-1a0ea730da92

📥 Commits

Reviewing files that changed from the base of the PR and between 7266439 and 30c1112.

📒 Files selected for processing (2)
  • extensions/kernel-debug-tiers.sh
  • extensions/sysrq-serial-trigger.sh

Comment thread extensions/sysrq-serial-trigger.sh
…_SEQUENCE

CR: ${SYSRQ_SERIAL_SEQUENCE:-sysrq} substitutes only when the variable
is unset, not when explicitly empty. An empty override silently writes
MAGIC_SYSRQ_SERIAL_SEQUENCE="" — which the kernel itself documents as
disabling BREAK-triggered SysRq entirely (see lib/Kconfig.debug). The
extension would then look enabled while doing nothing on the wire.

Treat empty the same way kconfig does: refuse it loudly so the operator
fixes their config instead of getting a silently-disabled SysRq path.

Signed-off-by: Igor Velkov <325961+iav@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

05 Milestone: Second quarter release Framework Framework components Needs review Seeking for review size/medium PR with more then 50 and less then 250 lines

Development

Successfully merging this pull request may close these issues.

1 participant