You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
machinelearning_run() 400 INVALID_ARGUMENT: GKE createTime passed verbatim to proto Timestamp (not RFC 3339) #2
machinelearning_run() fails with HTTP 400 INVALID_ARGUMENT at the control-plane create POST when the GKE workload createTime is not strict RFC 3339. The SDK passes workload_details["creation-timestamp"] verbatim into the google.protobuf.Timestamp field workload_details.gke.create_time.
POST .../machineLearningRuns?machine_learning_run_id=<id>
400 INVALID_ARGUMENT
fieldViolations[0]:
field: machine_learning_run.workload_details.gke.create_time
description: "Invalid value at '...gke.create_time' (google.protobuf.Timestamp),
Field 'createTime', Illegal timestamp format; timestamps must end with 'Z'
or have a valid timezone offset."
creation_timestamp=workload_details.get("creation-timestamp")
ifcreation_timestamp:
gke_workload_details["createTime"] =creation_timestamp# verbatim → proto Timestamp
No RFC-3339 normalization. The source creation-timestamp (from GKE_DIAGON_METADATA) isn't guaranteed to be RFC 3339. Also host_utils.get_identifier itself formats timestamps as "%Y%m%d-%H%M%S" (no Z), so this is latent even for intended inputs.
Fix
PR #1 adds a _normalize_rfc3339() helper applied at the createTime assignment (best-effort; unparseable values pass through). Verified: the 400 cleared and the MLRun was created.
— Reported via Navi on behalf of @lokic233 (Meta MRS-CE).
Activity
lokic233 commentedon Jun 20, 2026
Proposed fix: #1 (adds
_normalize_rfc3339(), verified clears this 400). — via Navi on behalf of @lokic233