Skip to content

Fix CodeQL scanning on GitHub and Azure DevOps - #401

Merged
Vladimir Morozov (vmoroz) merged 7 commits into
microsoft:mainfrom
vmoroz:codeql-advanced-setup
Nov 1, 2026
Merged

Vladimir Morozov (vmoroz) merged 7 commits into
microsoft:mainfrom
vmoroz:codeql-advanced-setup

Conversation

@vmoroz

@vmoroz Vladimir Morozov (vmoroz) commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

The Security overview reports a code scanning configuration error. CodeQL
default setup fails for five of its seven languages:

  • JavaScript/TypeScript: the extractor hits a StackOverflowError on the
    deliberately deep test/Parser/nested-expressions.js parser test.
  • C/C++: the analysis runs out of memory on a standard runner, mostly on
    vendored code in external/. Default setup then dropped C/C++, so the
    engine is not being scanned on GitHub.
  • Swift, Ruby, Java/Kotlin: there is no project or code CodeQL can build.

The same parser test also breaks the interpreted-language scan in the Azure
DevOps pipelines: CodeQL 3000 builds one database for JavaScript, Python,
PowerShell and Actions, and the crash fails all of them. CodeQL 3000 has no
documented setting that changes what is extracted, but the extractors read
LGTM_INDEX_FILTERS from the environment. The C++ scan in Azure DevOps is
unaffected.

  • GitHub workflow. .github/workflows/codeql.yml scans Actions, C/C++,
    JavaScript/TypeScript and Python without a build, on pushes and PRs to
    main, weekly and on demand. It keeps the /language:<lang> categories
    so existing alerts carry over.
  • GitHub scope. .github/codeql/codeql-config.yml excludes external,
    test, unittests, benchmarks, .yarn and **/__tests__. This avoids
    the extractor crash and keeps C/C++ within runner memory.
  • Azure DevOps scope. .ado/codeql-variables.yml sets
    LGTM_INDEX_FILTERS to exclude the same paths as the GitHub config. The
    CI, release and PR pipelines include it; traced C++ builds ignore it.
  • Azure DevOps classification. CodeQL.yml marks vendored code as
    library and tests and benchmarks as test, so CodeQL 3000 does not file
    their findings as SDL bugs. It does not change what is extracted.
  • Docs. AGENTS.md notes where both setups live and that GitHub default
    setup must stay off.

After merge, GitHub default setup must be switched off in Settings → Advanced
Security, and the stale Swift, Ruby, Java/Kotlin and 2022
codeql-analysis.yml configurations deleted from the CodeQL status page.

Test Plan

  • actionlint .github/workflows/codeql.yml (v1.7.12): passed.

  • All three YAML files parse with PyYAML; every CodeQL.yml path exists.

  • git diff --check: passed.

  • The first PR run analyzed all four languages; C/C++ finished in 28 minutes
    on a standard runner without running out of memory. Every job failed only
    at upload, because GitHub rejects advanced-setup results while default
    setup is enabled. They should pass once default setup is switched off.

  • Local CodeQL CLI 2.27.0 on a clean checkout, using the CodeQL 3000
    commands from the CI log (database init --build-mode none --db-cluster,
    database trace-command --use-build-mode, database finalize):

    • without filters: StackOverflowError on
      test/Parser/nested-expressions.js, exit code 2 (same as CI);
    • with LGTM_INDEX_FILTERS: JavaScript, Python and Actions databases
      finalize; no files from the excluded folders; lib/InternalJavaScript,
      API/hermes/extensions, .ado/scripts and workflows are included; no
      query-pack downloads.
  • A temporary step in this PR's Azure build printed all six
    LGTM_INDEX_FILTERS lines intact; it has been removed.

  • CodeQL 3000 does not run on pull branches, so the CI scan of main is the
    first real run; its cadence allows one after 2026-11-02 22:15 UTC.

No engine build or tests were run: only CI configuration and docs changed.

Default setup failed for five languages: the JavaScript extractor
overflows its stack on a deep parser test, C/C++ runs out of memory on
vendored code, and Swift, Ruby and Java/Kotlin have nothing it can build.

Scan Actions, C/C++, JavaScript/TypeScript and Python without a build,
and exclude external/, tests and benchmarks from analysis.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 30, 2026 23:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer validation is needed because enabled default setup blocks SARIF processing, while the still-running C/C++ analysis leaves successful completion, coverage, and memory usage unverified.

No open issues 🎉

What changed in this PR

Moves Hermes Windows code scanning to CodeQL advanced setup to address extraction failures and restore C/C++ scanning while excluding vendored code and test fixtures.

Changes:

  • Adds no-build scans for Actions, C/C++, JavaScript/TypeScript, and Python on pushes, pull requests, and a weekly schedule.
  • Configures path exclusions to avoid problematic fixtures and reduce analysis memory usage.
  • Documents the requirement to keep default setup disabled so workflow uploads are accepted.
File Description
AGENTS.md Documents the CodeQL configuration locations and default-setup restriction.
.github/​workflows/​codeql.yml Adds the four-language scanning workflow and preserves existing alert categories.
.github/​codeql/​codeql-config.yml Excludes vendored code, tests, benchmarks, and Yarn-managed files from scanning.

Review effort: Balanced

The no-build C/C++ analysis takes about 30 minutes. Run it on pushes to
main, weekly and on demand; PRs scan only the fast languages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 31, 2026 00:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The migration requires repository-admin changes outside this diff, and all reported uploads were rejected, leaving end-to-end scanning and alert continuity unverified.

No open issues 🎉

Review effort: Balanced

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Benchmark Results

Total benchmarks: 204

Inputs: baseline/baseline.json, bench_result.json

v8 (6176ms, 5005ms) -19.0%
v8 hermes (CI) hermes (CI)
v8-crypto 622.6ms 398.6ms
v8-deltablue 739.4ms 601.2ms
v8-raytrace 127.8ms 89.6ms
v8-regexp 593ms 519.8ms
v8-richards 937ms 697ms
v8-splay 225.2ms 200.6ms
v8-crypto (static) 452ms 295ms
v8-deltablue (static) 661.6ms 543.2ms
v8-raytrace (static) 91.8ms 78.4ms
v8-regexp (static) 749.2ms 715ms
v8-richards (static) 781.4ms 670.8ms
v8-splay (static) 195ms 196ms
test-suites (317886ms, 234219ms) -26.3%
test-suites hermes (CI) hermes (CI)
box2d 2949.8ms 1834ms
earley-boyer 2564.8ms 1778.4ms
navier-stokes 5743.8ms 3259ms
pdfjs 1038.2ms 768.8ms
gbemu 2296ms 1732.8ms
code-load 4472.8ms 4359.8ms
typescript 3381.8ms 2629.8ms
simpleSum 8855.4ms 5656.6ms
propAccess 2691.2ms 1892.8ms
allocObj 252ms 187.4ms
allocObjLit 6371.8ms 4506.8ms
allocNewObj 20053ms 14021.6ms
allocArray 249ms 184.8ms
allocNewArray 34956.8ms 23620.2ms
arrayRead 115.6ms 73.2ms
arrayReadByIndex 550ms 281.8ms
largeArrayRead 827.2ms 406.2ms
arrayWrite 261.8ms 166.8ms
largeArrayWrite 2218ms 1372.8ms
interp-dispatch 2880ms 1778.2ms
wb-perf 8327ms 4967.8ms
arrayReverse 40.6ms 31.2ms
arrayMap 1472.6ms 1007.4ms
arrayIndexOf 143.2ms 120.2ms
arrayLastIndexOf 148.6ms 122.6ms
arrayEvery 2308.8ms 1439.8ms
arraySome 2174.4ms 1445.4ms
arrayFill 2401.2ms 1616.2ms
arrayFilter 1866.4ms 1256.4ms
arrayFind 3207ms 2354.6ms
arrayFindIndex 3322ms 2371.2ms
arrayPop 1031.4ms 689.4ms
arrayReduce 1994.2ms 1370.4ms
arrayReduceRight 2075.8ms 1375ms
arrayShift 2065.8ms 1369.8ms
arrayUnshift 2109.4ms 1377.8ms
arrayIncludes 1216.6ms 769.4ms
arrayFrom 1102.8ms 853ms
arrayCopyWithin 1487ms 989.2ms
stringFromCharCode 92.6ms 62.4ms
arraySlice 786.2ms 583.4ms
arraySplice 26.8ms 28.2ms
arrayOf 1048.8ms 681.4ms
stringCharAt 1346.6ms 849.2ms
stringMatch 3192.4ms 2221.4ms
stringSearch 3508.2ms 2207.4ms
stringStartsWith 701ms 398.6ms
stringEndsWith 650.4ms 358.6ms
stringIncludes 1520.6ms 1394.2ms
stringIndexOf 1555.6ms 1707ms
stringLastIndexOf 1821.4ms 1599.8ms
stringSplit 822ms 547.8ms
stringSlice 490ms 372.4ms
stringPadStart 2910.8ms 2043.4ms
stringPadEnd 2910ms 2054.6ms
regExpMatch 1506.2ms 1177.4ms
regExpSearch 1227ms 1281ms
regExpToString 1204ms 885.6ms
stringReplace 1394.8ms 1537ms
regExpReplace 3309.2ms 667ms
regExpFlags 956.8ms 657.4ms
regExpSplit 1307.8ms 953.8ms
numberArrayReadWrite 2477.8ms 1723ms
protoCache 3594.6ms 1748.4ms
box2d (static) 1792.6ms 1458.4ms
earley-boyer (static) 1875.6ms 1519ms
navier-stokes (static) 3410.2ms 2533.4ms
pdfjs (static) 812.8ms 631.8ms
gbemu (static) 1797ms 1324.2ms
code-load (static) 4199.2ms 4076ms
typescript (static) 2685.6ms 2432.4ms
simpleSum (static) 936.2ms 648.8ms
propAccess (static) 2298ms 2032.6ms
allocObj (static) 0.2ms 0.4ms
allocObjLit (static) 4154.4ms 3503.8ms
allocNewObj (static) 15749.2ms 12158.4ms
allocArray (static) 0.6ms 0.6ms
allocNewArray (static) 28494.2ms 21539.8ms
arrayRead (static) 74.2ms 53.6ms
arrayReadByIndex (static) 453.2ms 359ms
largeArrayRead (static) 552.2ms 434.2ms
arrayWrite (static) 178.4ms 130ms
largeArrayWrite (static) 1568.6ms 1119ms
interp-dispatch (static) 1918.8ms 1475.4ms
wb-perf (static) 8991.2ms 7116.4ms
arrayReverse (static) 37ms 29.6ms
arrayMap (static) 1064.8ms 847.4ms
arrayIndexOf (static) 128.2ms 120.8ms
arrayLastIndexOf (static) 138.8ms 119.8ms
arrayEvery (static) 1397.6ms 1101.4ms
arraySome (static) 1406.6ms 1158.4ms
arrayFill (static) 1969.2ms 1515.6ms
arrayFilter (static) 1061ms 826ms
arrayFind (static) 2242.2ms 1871ms
arrayFindIndex (static) 2235.8ms 1914.6ms
arrayPop (static) 878ms 652.6ms
arrayReduce (static) 1311.8ms 1126ms
arrayReduceRight (static) 1322.6ms 1146.2ms
arrayShift (static) 1512ms 1198ms
arrayUnshift (static) 1568.2ms 1218.4ms
arrayIncludes (static) 963.4ms 733.8ms
arrayFrom (static) 1030.6ms 749.6ms
arrayCopyWithin (static) 1109.6ms 874.2ms
stringFromCharCode (static) 77ms 54.8ms
arraySlice (static) 571.6ms 497ms
arraySplice (static) 26ms 27ms
arrayOf (static) 868.8ms 631.4ms
stringCharAt (static) 1100.6ms 752.2ms
stringMatch (static) 2227.6ms 2223.4ms
stringSearch (static) 2219.2ms 2217ms
stringStartsWith (static) 494.2ms 381.8ms
stringEndsWith (static) 457.8ms 337.2ms
stringIncludes (static) 1437.4ms 1366ms
stringIndexOf (static) 1440.8ms 2272.8ms
stringLastIndexOf (static) 1956.6ms 1915.4ms
stringSplit (static) 651.2ms 519ms
stringSlice (static) 445.6ms 344.4ms
stringPadStart (static) 2294.4ms 1870.4ms
stringPadEnd (static) 2499.2ms 1882.8ms
regExpMatch (static) 1483.2ms 1114.2ms
regExpSearch (static) 1296.2ms 1227.4ms
regExpToString (static) 1326.6ms 1039.6ms
stringReplace (static) 1455.6ms 1565.2ms
regExpReplace (static) 851.8ms 623ms
regExpFlags (static) 870.6ms 593ms
regExpSplit (static) 1200ms 878ms
numberArrayReadWrite (static) 2132.8ms 1491ms
protoCache (static) 3595.8ms 2865.4ms
micros (60876ms, 55501ms) -8.8%
micros hermes (CI) hermes (CI)
getNodeById.js 5273ms 3441.4ms
setInsert.js 2864.8ms 1927ms
stringify-number.js 1823.4ms 1287.8ms
typed-array-sort.js 22381ms 22055.8ms
getNodeById.js (static) 3941ms 2974.2ms
setInsert.js (static) 2517.6ms 1855ms
stringify-number.js (static) 1663.6ms 1123.8ms
typed-array-sort.js (static) 20411.4ms 20835.6ms
jit-benches (8563ms, 5650ms) -34.0%
jit-benches hermes (CI) hermes (CI)
idisp.js 2810.2ms 1784.4ms
idispn.js 3395.8ms 2108ms
idisp.js (static) 1906ms 1469.2ms
idispn.js (static) 450.8ms 288.4ms
many-subclasses (73437ms, 54542ms) -25.7%
many-subclasses hermes (CI) hermes (CI)
many.js 19104.8ms 15423ms
many-sh-1.js 7322.8ms 5158.8ms
many-sh-2.js 7329ms 5173.2ms
many-sh-3.js 7083.6ms 4959.6ms
many-sh-4.js 7235ms 4989.6ms
many.js (static) 15996ms 11432.6ms
many-sh-1.js (static) 2285.2ms 1738.2ms
many-sh-2.js (static) 2387ms 2021.2ms
many-sh-3.js (static) 2347.2ms 1826.2ms
many-sh-4.js (static) 2346ms 1819.2ms
map-objects (3627ms, 2754ms) -24.1%
map-objects hermes (CI) hermes (CI)
map-objects-untyped.js 1023.6ms 741.6ms
map-objects-typed.js 957.6ms 694ms
map-objects-untyped.js (static) 917.2ms 726ms
map-objects-typed.js (static) 728.8ms 592.8ms
map-strings (4335ms, 3542ms) -18.3%
map-strings hermes (CI) hermes (CI)
map-strings-untyped.js 1196.8ms 952.2ms
map-strings-typed.js 1139.2ms 902.6ms
map-strings-untyped.js (static) 1092.2ms 922ms
map-strings-typed.js (static) 906.8ms 765ms
nbody (3448ms, 2133ms) -38.2%
nbody hermes (CI) hermes (CI)
original/nbody.js 839.6ms 478ms
fully-typed/nbody.js 710ms 378.8ms
fully-typed/nbody.ts 865.2ms 476.4ms
original/nbody.js (static) 449.2ms 347.4ms
fully-typed/nbody.js (static) 135ms 106.6ms
fully-typed/nbody.ts (static) 449.2ms 345.4ms
string-switch (6503ms, 5242ms) -19.4%
string-switch (string-switch/plain) hermes (CI) hermes (CI)
bench.js 1316ms 952.4ms
bench.js (static) 5187ms 4289.8ms
raytracer (5608ms, 4379ms) -21.9%
raytracer (raytracer/original) hermes (CI) hermes (CI)
bench-raytracer.js 1550.4ms 1161.2ms
raytracer.ts 1667.6ms 1182.8ms
bench-raytracer.js (static) 1198.4ms 1017ms
raytracer.ts (static) 1191.8ms 1018.2ms
MiniReact (30332ms, 24180ms) -20.3%
MiniReact hermes (CI) hermes (CI)
no-objects/out/simple-stripped.js 2239.2ms 1716ms
no-objects/out/simple-lowered.js 2270.8ms 1756ms
no-objects/out/music-stripped.js 42.6ms 39.4ms
no-objects/out/music-lowered.js 47.2ms 42.4ms
no-deps/stripped/MiniReact.js 5004ms 3671.8ms
no-deps/MiniReact.js 5256.2ms 3628.8ms
no-objects/out/simple.js 2260.4ms 1722.8ms
no-objects/out/music.js 45.6ms 41.4ms
no-objects/out/simple-stripped.js (static) 1725.2ms 1489.2ms
no-objects/out/simple-lowered.js (static) 1725.4ms 1578.4ms
no-objects/out/music-stripped.js (static) 18.8ms 19.2ms
no-objects/out/music-lowered.js (static) 19.4ms 19.4ms
no-deps/stripped/MiniReact.js (static) 4024.8ms 3509.8ms
no-deps/MiniReact.js (static) 3938.2ms 3441.2ms
no-objects/out/simple.js (static) 1695.2ms 1484.6ms
no-objects/out/music.js (static) 19ms 19.6ms
widgets (12934ms, 9829ms) -24.0%
widgets hermes (CI) hermes (CI)
simple-classes/widgets.js 1758.2ms 1133.4ms
original/es5/widgets.js 2798.8ms 2061.2ms
single-file/es5/widgets.js 2775.2ms 2060.6ms
simple-classes/widgets.js (static) 1012ms 847.2ms
original/es5/widgets.js (static) 2283.2ms 1857.6ms
single-file/es5/widgets.js (static) 2306.2ms 1869.4ms

@vmoroz
Vladimir Morozov (vmoroz) marked this pull request as draft October 31, 2026 02:58
Run all four CodeQL languages on pull requests again; the C/C++ scan
takes about as long as the Azure PR build.

Add CodeQL.yml so the Azure DevOps CodeQL 3000 scan classifies vendored
code as library and tests and benchmarks as test, keeping their findings
out of SDL bug filing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@vmoroz Vladimir Morozov (vmoroz) changed the title Move CodeQL to advanced setup with first-party paths Fix CodeQL scanning on GitHub and classify CodeQL 3000 test findings Oct 31, 2026
The JavaScript extractor overflows its stack on the deeply nested
expressions in test/Parser/nested-expressions.js, which fails the whole
interpreted-language database (JavaScript, Python, PowerShell, Actions)
in the 1ES SDL Sources job. CodeQL 3000 offers no documented setting
that changes what is extracted, so set LGTM_INDEX_FILTERS, which the
interpreted-language extractors read from the environment, in a shared
variables template used by the CI, release and PR pipelines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Revert before merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CodeQL 3000 does not run on pull branches, so check instead that the
multi-line pipeline variable reaches job tasks intact. Revert before
merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The PR build showed all six filter lines reach job tasks intact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@vmoroz Vladimir Morozov (vmoroz) changed the title Fix CodeQL scanning on GitHub and classify CodeQL 3000 test findings Fix CodeQL scanning on GitHub and Azure DevOps Oct 31, 2026
@vmoroz
Vladimir Morozov (vmoroz) marked this pull request as ready for review October 31, 2026 19:04
Copilot AI balanced review requested due to automatic review settings October 31, 2026 19:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The fix relies on an undocumented extraction filter and injected Azure DevOps tasks unavailable here, while main-branch scanning and GitHub uploads still require end-to-end validation after disabling default setup.

No open issues 🎉

Review effort: Balanced

@vmoroz
Vladimir Morozov (vmoroz) merged commit bef618e into microsoft:main Nov 1, 2026
20 of 23 checks passed
@vmoroz
Vladimir Morozov (vmoroz) deleted the codeql-advanced-setup branch November 1, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants