[pull] main from ollama:main - #835
Merged
Merged
Conversation
The runner package used to pick an engine from the first argument of the runner subcommand. Only the MLX engine is left, so the dispatcher has a single arm, its README still describes the removed Go runner's flags and endpoints, and the standalone cmd/runner binary exists only to invoke it. We call mlxrunner.Execute directly from the hidden runner subcommand and drop the --mlx-engine argument from the command line the MLX client spawns. Both sides ship in the same binary, so nothing has to accept both forms. The runner package and cmd/runner are removed. The subcommand's help hook hands the runner a bare --help. cobra calls the hook with no arguments for `ollama help runner`, which used to index past the end of the slice.
Several pieces outlived the code that used them. The root tokenizer package implemented the GGUF-side vocabularies for the Go engine and the safetensors-to-GGUF converter; nothing has imported it since the converter went. ml/backend.go held the Go engine's Backend, Context and Tensor interfaces, with fs.Config existing only to be returned from them, and a single CUDA template instance under ml/backend/ggml survived the engine removal along with the gitattributes entries for that tree and the CI change-filter globs for it and for the long-gone llama/llama.cpp. From the image generation engine, an integration test group that no test registers, its build tag, and the StepBar progress widget remained. DeviceInfo.IsBetter has no caller at all. All of it goes. Tidying the module file drops the regexp2 dependency and leaves protobuf as an indirect requirement. The llama3.2 tokenizer fixtures stay: the MLX runner's tokenizer uses them for its GGML parity test.
The MLX runner is the only Go inference runner left and is no longer experimental, so its packages leave x/. The bindings become a top-level mlx package beside the carried patches in mlx/compat, mirroring how llama/ holds the llama.cpp integration, and the runner becomes mlxrunner with the architectures nested under the package they implement. Subpackages move with their parent unless listed. x/mlxrunner/mlx mlx x/internal/mlxthread mlx/mlxthread x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest x/internal/mlxtest mlx/mlxtest x/quant mlx/quant mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx) x/mlxrunner mlxrunner x/models/nn mlxrunner/nn x/models/<arch> mlxrunner/model/<arch> x/mlxrunner/imports.go mlxrunner/model/architectures (new package) x/create create x/safetensors fs/safetensors x/tokenizer mlxrunner/tokenizer Every package keeps its name, so the Go changes are the import path rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift check and Darwin payload script follow the new paths. Four edits are not paths: the runner's blank architecture imports become the package mlxrunner/model/architectures, so the list to extend for a new model sits beside the architecture directories; a depguard rule keeps the two test harnesses out of non-test code, as the x/internal placement used to; the CI change filter's two entries for the long-deleted x/imagegen/mlx now name the bindings' CMake project and the carried patches, so a change to either builds the payload; and the tokenizer parity test reads its fixtures from its own testdata instead of walking out of x/. x/server and x/imagegen/manifest stay for the next two commits.
model/base held the Model interface and the architecture registry while model held weight loading and quant parameters, and every architecture imported both. create/client was the CLI side of safetensors imports, with the create command as its only caller, a duplicate of the command's adapter error, and a name that read like a second API client. The safetensors show helpers had their own package under x/ although it already declared package server. base merges into model, so base.Model and base.Register become model.Model and model.Register; nothing in the two overlapped. The create client's two files and their tests join package cmd, and the five names the command called are no longer exported. The show helpers join the server package, and the three entry points routes.go calls become unexported like the helpers around them.
The runner and its weight loader read a model's manifest through x/imagegen/manifest, the last piece of the removed image generation engine. It was a hand-rolled copy of the manifest package: its own model name parser with the default registry and namespace spelled out, its own blob path builder, and re-spelled media types, plus a model_index.json reader and other helpers that nothing has called since the engine went. The manifest package gains the three lookups the runner needs, a config layer by path, its contents, and the tensor layers, and ReadConfigJSON is built on the second of them. The weight loader resolves the model name with the shared parser, which fills in the same defaults the copy did, and locates blobs with BlobsPath. The architectures' calls to read their config.json compile unchanged. x/imagegen is gone.
nn.go held every layer type in the package apart from attention, recurrence and rope: the Linear and Embedding interfaces with their dense and quantized types, Conv1d, RMSNorm, LayerNorm and MultiLinear, with one test file to match. Finding a layer meant scanning the file named after the package. Each layer kind gets its own file: linear.go and embedding.go hold the interface and the dense and quantized types, conv.go, norm.go and multilinear.go take the rest, and nn_test.go splits the same way. The Layer and MultiLinearLayer interfaces go; nothing implemented or accepted them. No code changes otherwise.
model is one package with three jobs: the contract between the runner and the architectures, the opened checkpoint, and building nn layers from checkpoint tensors. Its files did not say which was which. base.go carried the folded package's name over the interfaces and the registry, root.go held the safetensors header scan next to Root, and quant.go mixed the nvfp4 global-scale helpers with quant parameter resolution. base.go becomes model.go, named for what it holds. root.go keeps Root and Open; TensorQuantInfo and the header scan join quant.go, so everything the checkpoint says about quantization is read and resolved in one file. The global-scale helpers move to globalscale.go with their tests. Root.Close, a no-op with one caller, goes. No code changes otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )