blob: add multipart uploads

[TuSKan]

Adds multipart upload to blob.

S3, GCS and Azure can all assemble one object from parts uploaded in any order, concurrently, or from separate processes. blob.Writer cannot express that — it is a single sequential stream owned by one goroutine — so today the only portable way to write a large object is to stream it whole, and callers who need the native behaviour drop out of the portable API into As.

API

u, err := b.NewMultipartUploader(ctx, key, nil)
part, err := u.UploadPart(ctx, blob.UploaderPart{Number: 1}, r)
err = u.Commit(ctx, parts)   // or u.Abort(ctx)

• Parts are numbered from 1 and assembled in ascending order regardless of the order they were uploaded in.
• UploadPart returns the part value Commit needs; the caller collects them.
• Nothing is readable at the key until Commit.
• UploadID is durable, so OpenMultipartUploader resumes an upload started in another process.
• MultipartUploaderOptions mirrors WriterOptions (content type, cache control, metadata, ContentCRC32C, BeforeUpload), minus the content-type sniffing, which a multipart upload cannot do.

Compatibility

Driver support is optional. blob.Bucket type-asserts driver.MultipartUploaderBucket and returns Unimplemented when the driver doesn't provide it, so no existing driver — in this repo or outside it — has to change. The prefixed and single-key wrappers forward the interface, so blob.PrefixedBucket and ?prefix= URLs keep multipart rather than reporting it unsupported.

Drivers

s3blob uses S3's native multipart upload. azureblob uses block blobs (StageBlock / CommitBlockList), gcsblob composes objects, and fileblob and memblob assemble the object themselves — fileblob by renaming the assembled file into place, so Commit stays atomic.

Testing

drivertest covers out-of-order upload, abort, resume, and that the committed object carries the content type, cache control and metadata it was created with. memblob and fileblob run it for real, including through the prefixed wrapper, and need no recordings. There's a runnable Example on memblob, so the feature is demonstrable without credentials.

The cloud drivers need replay recordings before this can go green. azureblob, gcsblob and s3blob test against recorded HTTP replays, and none exist for the new multipart calls, so the new conformance subtest fails for them:

--- FAIL: TestConformance/TestMultipartUploader
    open testdata\TestConformance\TestMultipartUploader.replay:
    The system cannot find the file specified.

Recording them needs real cloud credentials, which I don't have.

How I'd suggest splitting this

The first commit — core API, memblob, conformance coverage, docs — builds and tests green on its own and needs no recordings. The second adds the four remaining drivers, three of which cannot pass CI until someone with credentials records them.

Happy to reshape along that line: land the core with memblob and fileblob (both fully tested), and let azureblob, gcsblob and s3blob follow in a separate PR once recordings exist. Or leave it as-is if you'd rather record them and take it in one go. Let me know which you prefer and I'll push it.

Worth arguing about

The exported surface: Bucket gains two methods and the package gains four types, and those are hard to change once released even at v0.x. I'd rather adjust the names and the shape of UploaderPart now than after.

[vangent]

Hmm. I don't think this is something we want or need to expose. The drivers should do this automatically.

For example, S3's transfermanager (which s3blob uses) says "UploadObject uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines." (https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager@v0.1.2#Options). Some of the behavior is managed by WriterOptions like MaxConcurrency and BufferSize. GCS supports BufferSize, but the driver doesn't (yet) use EnableParallelUpload (https://pkg.go.dev/cloud.google.com/go/storage#hdr-Parallel_Uploads).

[TuSKan]

Fair enough on throughput — you're right, it plainly can, and BufferSize/MaxConcurrency already tune it. s3blob wires them into transfermanager, azureblob into UploadStream.

Three things I'd still like your read on.

gcsblob is the odd one out today. It sets ChunkSize from BufferSize but never reads MaxConcurrency, and doesn't use EnableParallelUpload. So of the three, GCS is the only one not getting the automatic behaviour.

Resumability isn't covered by any of them. transfermanager.Client exposes UploadObject and no upload ID, no resume, no abort handle. If the process dies mid-upload the transfer starts from zero, and the incomplete MPU is orphaned in S3 until a lifecycle rule aborts it. Is that a gap you'd consider in scope for blob, or deliberately out of it?

That last clause is the crux for me. When the driver owns the upload ID, a crash leaves state the caller cannot see, resume, or abort; recovery is only possible out-of-band via lifecycle rules. When the caller owns it, they can persist it and either finish or abort on restart.

So perhaps the real question isn't multipart vs. automatic, but whether blob wants to expose any handle on in-flight upload state at all. If the answer is no, I'm happy to close this. no problem.

[vangent]

"gcsblob is the odd one out today" -> yes, let's deal with that explicitly.

"whether blob wants to expose any handle on in-flight upload state" -> I think the answer to that is "no". I'd prefer to keep the simple semantics we have, which seem to be good enough for ~everyone (nobody has asked for this feature in the last few years). Escape hatches exist for users who want to handle the edge cases.