Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/cd-llvm-firtool.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@ name: Continuous Delivery of llvm-firtool

on:
workflow_dispatch:
# Run every 15 min, checking for new releases is cheap.
schedule:
- cron: '*/15 * * * *'
Comment thread
jackkoenig marked this conversation as resolved.
Resolved


jobs:
Expand Down
70 changes: 67 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,73 @@
# Firtool Resolver and LLVM Firtool Native

Firtool Resolver is a Scala library and command-line application for downloading [LLVM Firtool](https://github.com/llvm/circt) as a Maven artifact.
LLVM Firtool Native is a project to distribute LLVM Firtool as Maven artifacts.
Firtool Resolver is a Scala library and command-line application for resolving and downloading [LLVM Firtool](https://github.com/llvm/circt).
It downloads binaries directly from CIRCT's GitHub releases and verifies their SHA-256 checksums.

# Firtool Native Dependency Distribution Specification (FNDDS)
LLVM Firtool Native (`org.chipsalliance:llvm-firtool`) and FNDDS are **legacy** distribution mechanisms. The last version published to Maven Central is **1.162.0**; existing artifacts and classpath overrides continue to work.

## Using Firtool Resolver

```scala
import firtoolresolver.Resolve

Resolve("1.163.0") match {
case Right(binary) => println(binary.path)
case Left(error) => sys.error(error)
}
```

The command-line interface is `firtool-resolver [-v] <version>`. For example:

```sh
cs launch --scala 2.13 org.chipsalliance::firtool-resolver:2.2.0 \
--main firtoolresolver.Main -- -v 1.163.0
Comment thread
jackkoenig marked this conversation as resolved.
Resolved
```

### Resolution order

1. Run `firtool` from the directory specified by `CHISEL_FIRTOOL_PATH`. An invalid path or a binary that cannot run is an unrecoverable error, as before.
2. Choose the effective version: system property `chisel.firtool.version`, then environment variable `CHISEL_FIRTOOL_VERSION`, then the caller's default version. An explicit override skips the classpath lookup and warns if `llvm-firtool` is also on the classpath.
3. If no explicit version is set, honor `llvm-firtool` on the classpath. This legacy mechanism logs a deprecation message at info level.
4. Reuse the cached binary at `<cache>/<effective-version>/bin/firtool`.
5. Download the effective version from GitHub, verify its checksum, and atomically install just the executable in the cache. Downloads are locked per version across processes.
6. As a last resort, fetch `llvm-firtool` with Coursier from Maven. This supports older versions and environments that can reach a Maven proxy but cannot reach GitHub; it only works for already-published artifacts.

### Configuration

| Setting | System property | Environment variable | Default |
|---|---|---|---|
| Executable directory | — | `CHISEL_FIRTOOL_PATH` | Unset |
| Version override | `chisel.firtool.version` | `CHISEL_FIRTOOL_VERSION` | Caller's default version |
| Release base URL / mirror | `chisel.firtool.url` | `CHISEL_FIRTOOL_URL` | `https://github.com/llvm/circt/releases/download` |
| Cache directory | — | `CHISEL_FIRTOOL_CACHE` | OS-specific project cache directory |

Properties take precedence over the corresponding environment variables.

Direct downloads require access to **github.com** and **release-assets.githubusercontent.com**. Standard JVM proxy properties (`https.proxyHost`, `https.proxyPort`, and the corresponding HTTP properties for HTTP mirrors) are honored.

To use an internal web server (HTTP or HTTPS), an Artifactory generic remote, or a local `file:` mirror, set the release base URL. The mirror must provide both the archive and its `.sha256` file at:

```text
<base>/firtool-<version>/firrtl-bin-<platform>.<ext>
<base>/firtool-<version>/firrtl-bin-<platform>.<ext>.sha256
```

| Resolver platform | Release asset |
|---|---|
| `linux-x64` | `firrtl-bin-linux-x64.tar.gz` |
| `macos-x64` | `firrtl-bin-macos-x64.tar.gz` |
| `macos-aarch64` | `firrtl-bin-macos-arm64.tar.gz` |
| `windows-x64` | `firrtl-bin-windows-x64.zip` |

GitHub fetching supports versions starting at **1.45.0**. Native macOS arm64 assets start at **1.144.0**; earlier versions fall back to macOS x64 through Rosetta. Older releases remain available through the legacy Maven path.

### Development and release

Run `./mill -i 'firtool-resolver[_].test'` for the hermetic MUnit resolver tests on both Scala versions, or `./mill -i 'firtool-resolver[2.13].test'` for just Scala 2.13.

The FileCheck integration suite is `./firtool-resolver/test/run_tests.sh <linux|macos|windows> <2.12|2.13>`; it requires Coursier, FileCheck, and network access, along with `LLVM_FIRTOOL_VERSION` and `LLVM_FIRTOOL_PRERELEASE` for the retained legacy `publishLocal` build.

# Firtool Native Dependency Distribution Specification (FNDDS) (legacy)

This specification follows [Semantic Versioning 2.0.0](https://semver.org).

Expand Down
15 changes: 14 additions & 1 deletion build.mill
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ object `llvm-firtool` extends JavaModule with ChipsAlliancePublishModule {

def firtoolVersion = getEnvVariable("LLVM_FIRTOOL_VERSION")
// FNDDS requires that the publish version start with firtool version with optional -<suffix>
def isSnapshot = T { getEnvVariable("LLVM_FIRTOOL_PRERELEASE") != "0" }
def isSnapshot = T.input { getEnvVariable("LLVM_FIRTOOL_PRERELEASE") != "0" }

def publishSuffix = T { if (isSnapshot()) "-SNAPSHOT" else "" }
def publishVersion = T { firtoolVersion + publishSuffix() }
Expand Down Expand Up @@ -293,6 +293,7 @@ trait FirtoolResolver extends CrossScalaModule with ChipsAlliancePublishModule {
def ivyDeps = Agg(
ivy"dev.dirs:directories:26",
ivy"io.get-coursier::coursier:2.1.8",
ivy"org.apache.commons:commons-compress:1.24.0",
)

// Modify the classpath to remove things we want to dynamically link (Scala jars).
Expand Down Expand Up @@ -334,4 +335,16 @@ trait FirtoolResolver extends CrossScalaModule with ChipsAlliancePublishModule {
Rule.Relocate(s"$p.**", s"firtoolresolver.shaded.$p.@1")
)
}

object test extends ScalaModule with TestModule.Munit {
def scalaVersion = root.scalaVersion
def scalacOptions = root.scalacOptions
def moduleDeps = Seq(core)
def ivyDeps = Agg(ivy"org.scalameta::munit:1.3.6")
// Mill loads tests through a separate classloader. Give subprocess tests
// the complete classpath explicitly rather than the test runner's launcher.
def forkArgs = T {
Seq("-Dfirtoolresolver.test.classpath=" + runClasspath().map(_.path.toString).mkString(java.io.File.pathSeparator))
}
}
}
94 changes: 94 additions & 0 deletions firtool-resolver/src/GitHub.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
// SPDX-License-Identifier: Apache-2.0

package firtoolresolver

import java.io.{FileNotFoundException, IOException, InputStream}
import java.net.{HttpURLConnection, URL}
import java.nio.file.{Files, Path, StandardCopyOption}
import java.security.{DigestInputStream, MessageDigest}
import java.util.Locale
import java.util.zip.GZIPInputStream
import scala.io.Source
import org.apache.commons.compress.archivers.ArchiveInputStream
import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
import org.apache.commons.compress.archivers.zip.ZipArchiveInputStream

/** Download CIRCT release assets and extract just the firtool executable. */
private[firtoolresolver] object GitHub {
private class NotFound(url: URL) extends IOException(s"Not found: $url")

/** A published release asset for one platform, with its expected SHA-256. */
final case class Release(version: String, platform: String, url: URL, sha256: String)

private def open(url: URL): InputStream = {
val connection = url.openConnection()
connection.setConnectTimeout(10000)
connection.setReadTimeout(60000)
connection.setRequestProperty("User-Agent", "firtool-resolver")
// HttpURLConnection follows same-protocol redirects (to release-assets.githubusercontent.com).
connection match {
case http: HttpURLConnection if http.getResponseCode == 404 => throw new NotFound(url)
case http: HttpURLConnection if http.getResponseCode != 200 =>
throw new IOException(s"HTTP ${http.getResponseCode} downloading $url")
case _ =>
}
try connection.getInputStream
catch { case _: FileNotFoundException => throw new NotFound(url) } // file: URLs
}

/** Look up the release asset and its checksum. None means the asset is not published for this platform.
*
* The checksum comes from the `<asset>.sha256` file published next to the asset, not from the `digest` field
* of the GitHub releases API, because:
* - `chisel.firtool.url` mirrors are plain file trees and do not implement the API
* - releases up to firtool-1.119.0 have `"digest": null`, while the .sha256 files exist for all versions
* - the API is rate limited (60 requests/hour unauthenticated), the download URLs are not
* - using the API would require parsing JSON
*/
def lookup(logger: Logger, version: String, platform: String): Option[Release] = {
val circtPlatform = if (platform == "macos-aarch64") "macos-arm64" else platform
val extension = if (platform.startsWith("windows-")) "zip" else "tar.gz"
val base = sys.props.get("chisel.firtool.url")
.orElse(sys.env.get("CHISEL_FIRTOOL_URL"))
.getOrElse("https://github.com/llvm/circt/releases/download")
val url = new URL(s"${base.stripSuffix("/")}/firtool-$version/firrtl-bin-$circtPlatform.$extension")
logger.debug(s"Downloading SHA-256 checksum from $url.sha256")
try {
val source = Source.fromInputStream(open(new URL(s"$url.sha256")), "UTF-8")
val hash = try source.mkString.trim.split("\\s+").head.toLowerCase(Locale.ROOT) finally source.close()
Some(Release(version, platform, url, hash))
} catch {
case _: NotFound =>
logger.debug(s"No GitHub release asset for firtool $version on $platform")
None
}
}

/** Stream the release archive, writing its firtool executable to destination, and verify the checksum. */
def extract(logger: Logger, release: Release, destination: Path): Unit = {
val url = release.url
val windows = release.platform.startsWith("windows-")
val name = if (windows) "firtool.exe" else "firtool"
logger.info(s"Downloading firtool ${release.version} for ${release.platform} from $url")
val digest = MessageDigest.getInstance("SHA-256")
val download = new DigestInputStream(open(url), digest)
val archive: ArchiveInputStream =
if (windows) new ZipArchiveInputStream(download) else new TarArchiveInputStream(new GZIPInputStream(download))
try {
logger.debug(s"Extracting firtool from $url")
Iterator.continually(archive.getNextEntry).takeWhile(_ != null)
.find(e => !e.isDirectory && (e.getName == s"bin/$name" || e.getName.endsWith(s"/bin/$name"))) match {
case Some(_) => Files.copy(archive, destination, StandardCopyOption.REPLACE_EXISTING)
case None => throw new IOException(s"$url does not contain bin/$name")
}
// Drain the rest of the download so the digest covers the whole archive.
val buffer = new Array[Byte](64 * 1024)
while (download.read(buffer) != -1) {}
} finally archive.close()
val actual = digest.digest().map(b => f"${b & 0xff}%02x").mkString
if (actual != release.sha256) {
throw new IOException(s"SHA-256 checksum mismatch for $url: expected ${release.sha256}, actual $actual")
}
logger.debug(s"Verified SHA-256 checksum: $actual")
}
}
Loading
Loading