Skip to content

Commit 1570c6e

Browse files
hebaalazzehparthea
andauthored
fix(ci): add matrix sharding to import profiler workflow (#17779)
## Overview This PR implements static matrix sharding for the `import-profiler` workflow to run modified packages in parallel across 8 shards, speeding up CI execution time on multi-package PRs. ## Key Changes * **Parallel Sharding**: Updated [`.github/workflows/import-profiler.yml`](file:///usr/local/google/home/hebaalazzeh/git/google-cloud-python/.github/workflows/import-profiler.yml) to split modified packages under `packages/` across 8 GitHub Actions runner matrix shards using modulo logic (`idx % TOTAL_SHARDS`). * **Unified Status Check**: Added an `all-import-profiles` aggregation job that depends on the matrix run (`needs: import-profile`) to serve as the single required GitHub status check for branch protection and merge queue rules. --------- Co-authored-by: Anthonios Partheniou <partheniou@google.com>
1 parent b8f6776 commit 1570c6e

1 file changed

Lines changed: 46 additions & 2 deletions

File tree

‎.github/workflows/import-profiler.yml‎

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,16 @@ jobs:
1616
import-profile:
1717
runs-on: ubuntu-latest
1818
timeout-minutes: 60
19+
strategy:
20+
fail-fast: false
21+
matrix:
22+
shard: [0, 1, 2, 3, 4, 5, 6, 7] # 8 parallel shards
23+
name: import-profile (Shard ${{ matrix.shard }})
1924
steps:
2025
- name: Checkout
2126
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
2227
with:
23-
fetch-depth: 2
28+
fetch-depth: 0 # Fetch git history to find changed packages
2429
persist-credentials: false
2530
- name: Setup Python
2631
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
@@ -41,5 +46,44 @@ jobs:
4146
PY_VERSION: "3.15"
4247
# Workaround: Allows libcst to compile on Python 3.15+ while PyO3 catches up
4348
PYO3_USE_ABI3_FORWARD_COMPATIBILITY: "1"
49+
SHARD_INDEX: ${{ matrix.shard }}
50+
TOTAL_SHARDS: 8
4451
run: |
45-
ci/run_conditional_tests.sh
52+
TARGET_BRANCH=${TARGET_BRANCH:-main}
53+
git fetch origin "${TARGET_BRANCH}" --deepen=200 || true
54+
55+
# Get unique list of modified packages under packages/
56+
modified_packages=$(git diff --name-only origin/"${TARGET_BRANCH}"... | grep '^packages/' | cut -d/ -f1,2 | sort -u)
57+
58+
# Filter packages assigned to this specific shard index
59+
idx=0
60+
packages_to_test=""
61+
for pkg in $modified_packages; do
62+
if [ -d "$pkg" ]; then
63+
if [ "$((idx % TOTAL_SHARDS))" -eq "${SHARD_INDEX}" ]; then
64+
packages_to_test="$packages_to_test $pkg"
65+
fi
66+
idx=$((idx + 1))
67+
fi
68+
done
69+
70+
# Run tests on the assigned packages
71+
if [ -n "$packages_to_test" ]; then
72+
echo "Shard ${{ matrix.shard }} running packages: $packages_to_test"
73+
PACKAGE_LIST="$packages_to_test" ci/run_conditional_tests.sh
74+
else
75+
echo "No packages assigned to Shard ${{ matrix.shard }}."
76+
fi
77+
78+
all-import-profiles:
79+
needs: import-profile
80+
if: always()
81+
runs-on: ubuntu-latest
82+
steps:
83+
- name: Check import profile results
84+
run: |
85+
if [[ "${{ needs.import-profile.result }}" != "success" && "${{ needs.import-profile.result }}" != "skipped" ]]; then
86+
echo "Import profiles failed"
87+
exit 1
88+
fi
89+
echo "All import profiles passed or were skipped"

0 commit comments

Comments
 (0)