Overlay: Use overlay-aware CLI version when analyzing PRs by henrymercer · Pull Request #3880 · github/codeql-action

henrymercer · 2026-05-06T18:18:36Z

When analyzing PRs, prefer CLI versions that have cached overlay-base databases to speed up analysis time. However to ensure we can effectively rollback new versions, do not use a CodeQL version whose feature flag is disabled, even if this means running without overlay analysis.

This will be shipped via two feature flags:

overlay_analysis_match_codeql_version_dry_run logs a diagnostic when the overlay-aware version differs from the latest enabled version
overlay_analysis_match_codeql_version uses the overlay-aware version when analysing PRs

Risk assessment

For internal use only. Please select the risk level of this change:

Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Workflow types:

Advanced setup - Impacts users who have custom CodeQL workflows.
Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.
Code Quality - The changes impact analyses when analysis-kinds: code-quality.
Other first-party - The changes impact other first-party analyses.

Environments:

Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.

How did/will you validate this change?

Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

Feature flags - All new or changed code paths can be fully disabled with corresponding feature flags.

How will you know if something goes wrong after this change is released?

Telemetry - I rely on existing telemetry or have made changes to the telemetry.
- Dashboards - I will watch relevant dashboards for issues after the release. Consider whether this requires this change to be released at a particular time rather than as part of a regular release.
- Alerts - New or existing monitors will trip if something goes wrong with this change.

Are there any special considerations for merging or releasing this change?

No special considerations - This change can be merged at any time.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Consider adding a changelog entry for this change.
Confirm the readme and docs have been updated if necessary.

Copilot

Pull request overview

This PR updates how the Action selects the default CodeQL CLI version so that, when analyzing pull requests, it can prefer an enabled CLI version that already has cached overlay-base databases for the configured languages (to speed up overlay/incremental analysis), while still respecting feature-flag rollback constraints.

Changes:

Extend the default CLI “version info” returned from feature flags to include a sorted list of enabled default versions (not just a single version).
Add PR-aware logic in setup-codeql to optionally pick the highest enabled version that intersects with overlay-base DB cache entries (with dry-run telemetry support).
Thread the new version-info shape through callers and update unit tests and the changelog entry.

Show a summary per file

File	Description
src/upload-lib.ts	Switches to the new `getEnabledDefaultCliVersions` API and passes `rawLanguages` through `initCodeQL`.
src/testing-utils.ts	Updates test fixtures/mocks for the new `CodeQLDefaultVersionInfo.enabledVersions` shape.
src/start-proxy.ts	Adapts to the new version-info shape by selecting `enabledVersions[0]` for proxy downloads.
src/start-proxy.test.ts	Updates stubbing to `getEnabledDefaultCliVersions`.
src/setup-codeql.ts	Implements overlay-aware default-version resolution for PR analyses (feature-flag gated) and threads `rawLanguages`.
src/setup-codeql.test.ts	Updates call sites for new `rawLanguages` parameter and adds unit tests for overlay-cache version filtering.
src/setup-codeql-action.ts	Updates to `getEnabledDefaultCliVersions` and passes `rawLanguages` (currently `undefined`).
src/init.ts	Threads `rawLanguages` through to `setupCodeQL`.
src/init-action.ts	Uses `getEnabledDefaultCliVersions` and passes `rawLanguages` derived from the `languages` input.
src/feature-flags.ts	Introduces `CodeQLVersionInfo`, changes default version info to `enabledVersions[]`, and adds new overlay match feature flags.
src/feature-flags.test.ts	Updates tests to validate multi-version enablement ordering/fallback behavior.
src/codeql.ts	Threads `rawLanguages` into tool setup to support PR-aware version resolution.
src/codeql.test.ts	Updates tests for the new default-version info shape and new function signatures.
CHANGELOG.md	Adds an UNRELEASED entry describing the experimental overlay-aware default version selection.
lib/upload-sarif-action-post.js	Generated JS output (not reviewed).
lib/upload-lib.js	Generated JS output (not reviewed).
lib/start-proxy-action.js	Generated JS output (not reviewed).
lib/start-proxy-action-post.js	Generated JS output (not reviewed).
lib/resolve-environment-action.js	Generated JS output (not reviewed).
lib/init-action.js	Generated JS output (not reviewed).
lib/autobuild-action.js	Generated JS output (not reviewed).
lib/analyze-action.js	Generated JS output (not reviewed).
lib/analyze-action-post.js	Generated JS output (not reviewed).

Copilot's findings

Comments suppressed due to low confidence (1)

src/start-proxy.test.ts:1029

The stub variable is named getDefaultCliVersion, but it actually stubs getEnabledDefaultCliVersions. Renaming the local variable (and the later assertion) would avoid confusion and better reflect what’s being tested.

      const getDefaultCliVersion = sinon
        .stub(features, "getEnabledDefaultCliVersions")
        .resolves({
          enabledVersions: [{ cliVersion: "2.20.1", tagName: expectedTag }],
        });
      const path = await startProxyExports.getProxyBinaryPath(logger, features);

      t.assert(getDefaultCliVersion.calledOnce);
      sinon.assert.calledOnceWithMatch(

Files reviewed: 14/26 changed files
Comments generated: 3

+      const version = await resolveDefaultCliVersion(
+        defaultCliVersion,
+        rawLanguages,
+        features,
+        logger,
+      );
+      cliVersion = version.cliVersion;
+      tagName = version.tagName;
    }


mbg

Thanks for breaking this up into (reasonably) sane commits and a separate PR from the previous one! Overall, this looks like a pretty good first stab at this change and I don't see any major issues here.

I have left a bunch of detailed comments. There are a couple in particular about possible issues down the road that we might want to consider documenting or guarding against.

I am wondering if the second FF is a bit overkill and adds an extra layer of complexity on top of an already fairly complex change. I don't feel strongly about this, but the thought crossed my mind whether a reasonable thing to do here to reduce complexity / risk would be to consider breaking this PR up into three separate changes:

The first commit that modifies some of the existing logic to return all FF-enabled CLI versions, but doesn't make use of this information. We can ship that change without a new FF.
Then ship the new logic that's guarded by the second FF. I.e. everything except actually using the CLI version that we determined using the new process.
Then ship the first FF + the logic that uses the CLI version determined using the new process.

mbg · 2026-05-07T14:52:13Z

+        enabledVersions: [
+          {
+            cliVersion: defaults.cliVersion,
+            tagName: defaults.bundleVersion,
+          },
+        ],


Minor: Could we define this as a constant to avoid some duplication?

mbg · 2026-05-07T15:00:25Z

    // version, or the one enabled by FFs.
    const versionInfo = useFeaturesToDetermineCLI
-      ? await getCliVersionFromFeatures(features)
+      ? (await getCliVersionFromFeatures(features)).enabledVersions[0]


This is an interesting change. It might mean that the bundle release used by start-proxy and init could diverge. That's certainly not a problem right now, since we don't depend on them lining up. (I.e. we have thus far not had any reason to require the proxy to be of at least a particular version for changes made in the CLI/extractors.)

It could also mean that a base database is extracted using the version of the proxy that shipped with the corresponding CLI, and we then end up using a newer proxy for the overlay analysis on a PR. I don't see an immediate problem with it, but it is worth noting.

We might want to think about whether this is something we want to document somehow so that it doesn't become a problem if we ever want to make that assumption.

mbg · 2026-05-07T15:04:23Z

      getTemporaryDirectory(),
      gitHubVersion.type,
      codeQLDefaultVersionInfo,
+      undefined, // rawLanguages: currently, setup-codeql is not language aware


A potential issue down the line if we ever want to expand on setup-codeql and allow it to run before init. It might be worth documenting the implications of this on overlay analysis here.

mbg · 2026-05-07T15:07:16Z

    const codeQLDefaultVersionInfo =
      await features.getEnabledDefaultCliVersions(gitHubVersion.type);
    toolsFeatureFlagsValid = codeQLDefaultVersionInfo.toolsFeatureFlagsValid;
+    const rawLanguages = configUtils.getRawLanguagesNoAutodetect(


Could we add a comment here to say what this is for? E.g. "Split the comma-separated input string (if any) into an array of its components to use for identifying overlay base databases".

mbg · 2026-05-07T15:22:51Z

+    sinon.stub(api, "getAutomationID").resolves("test/");
+    sinon.stub(api, "listActionsCaches").resolves([
+      {
+        key: "codeql-overlay-base-database-1-aaaaaaaaaaaaaaaa-javascript-2.20.2-abc-1-1",


Minor: Could we recycle one of the functions for computing the cache key here so that this doesn't break immediately if we make any changes to the cache key format?

mbg · 2026-05-07T15:25:38Z

+      {
+        key: "codeql-overlay-base-database-1-aaaaaaaaaaaaaaaa-javascript-2.20.1-def-2-1",
+      },
+      {
+        key: "codeql-overlay-base-database-1-aaaaaaaaaaaaaaaa-javascript-2.20.0-ghi-3-1",
+      },


Swap the order of these to exercise the "sorted desc" part of the test?

mbg · 2026-05-07T15:27:32Z

+
+    const result = await setupCodeql.getEnabledVersionsWithOverlayBaseDatabases(
+      overlayMatchEnabledVersions,
+      undefined,


Test name says "empty", but here we pass in undefined directly. What if we get []?

mbg · 2026-05-07T15:28:19Z

+);
+
+test.serial(
+  "getEnabledVersionsWithOverlayBaseDatabases includes the highest version when it is cached",


Could we make this test name a bit clearer? The test body seems to just set up a single cache and the test then checks that it is included in the results. I don't see how this is specific to "the highest"?

mbg · 2026-05-07T15:36:38Z

+    const useOverlayAwareDefaultCliVersion = !!analysisKinds?.includes(
+      AnalysisKind.CodeScanning,
+    );


Just to be clear, is the intention to:

Use this for only code-scanning analyses?

Or for any analysis involving code-scanning?

Co-authored-by: Michael B. Gale <mbg@github.com>

henrymercer added 5 commits May 6, 2026 15:14

Add OverlayAnalysisMatchCodeqlVersion feature flag

a796e3e

Expose all enabled default CLI versions

b094211

Match CLI version to cached overlay-base database

55d6319

Add dry run mode so we can dark ship

b967fdf

Add changelog note

1b56327

Copilot AI review requested due to automatic review settings May 6, 2026 18:18

henrymercer requested a review from a team as a code owner May 6, 2026 18:18

Copilot started reviewing on behalf of henrymercer May 6, 2026 18:19 View session

Merge branch 'main' into henrymercer/overlay-match-codeql-version

817b684

github-actions Bot added the size/XL May be very hard to review label May 6, 2026

Copilot AI reviewed May 6, 2026

View reviewed changes

henrymercer added 2 commits May 7, 2026 11:00

Filter to code scanning only

01bc9be

Nit: Dedupe languages

7525c68

mbg reviewed May 7, 2026

View reviewed changes

Improve changelog note

efc9b0a

Co-authored-by: Michael B. Gale <mbg@github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overlay: Use overlay-aware CLI version when analyzing PRs#3880

Overlay: Use overlay-aware CLI version when analyzing PRs#3880
henrymercer wants to merge 9 commits intomainfrom
henrymercer/overlay-match-codeql-version

henrymercer commented May 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

mbg left a comment

Uh oh!

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

mbg May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

henrymercer commented May 6, 2026

Risk assessment

Which use cases does this change impact?

How did/will you validate this change?

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

How will you know if something goes wrong after this change is released?

Are there any special considerations for merging or releasing this change?

Merge / deployment checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

mbg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants