How to prevent duplicated timer execution when deploying new Ergo nodes in Kubernetes?

[LIGUANGYAO]

Hello Ergo team,

We are currently using Ergo to build a game server system, and we are running our services in a Kubernetes environment.

We have encountered a design problem regarding Actor timers and rolling deployments.

Our current architecture is roughly:

• Each player/game entity is represented by an Actor.
• The Actor maintains some runtime state in memory.
• The Actor can create timers for scheduled tasks, such as:
  • player offline cleanup
  • periodic state synchronization
  • game activities
  • scheduled rewards
  • delayed actions

The problem appears when we deploy a new version using Kubernetes.

During a rolling update:

Old Node
   |
   |  Actor A
   |  Timer running
   |
   v

New Node
   |
   |  Actor A restarted/migrated
   |  Timer created again

Now we may have two timers running for the same logical Actor:

Old Actor Timer
        |
        v
      execute


New Actor Timer
        |
        v
      execute

This can cause duplicated execution of business logic.

For example:

• a scheduled reward may be granted twice
• a timeout event may trigger twice
• periodic jobs may run simultaneously on multiple nodes

We are trying to understand the recommended approach in the Actor model.

Some possible solutions we are considering:

1. Store timer information externally and recreate timers after Actor restart.

Example:

Actor state
+
Timer metadata
        |
        v
Database / Redis

After restart:

Actor Init()
    |
    v
Load timer state
    |
    v
Create new timer

2. Use distributed locks or leader election.
   For example:

Actor Timer Trigger

        |
        v

Redis/Etcd lock

        |
        v

Only one node executes

3. Use a durable queue / scheduler service.

Example:

Timer Event

    |
    v

Persistent Queue

    |
    v

Actor consumes event

However, we are not sure which approach fits best with the Ergo design philosophy.

My questions are:

1. In Ergo, what is the recommended way to handle timers for long-lived Actors in a distributed environment?
2. Should Actor timers be considered only local/in-memory and not responsible for reliable execution?
3. For Kubernetes rolling deployment, how should we prevent old and new Actor instances from executing the same scheduled task?
4. Is there a recommended pattern similar to Erlang/OTP supervision, distributed processes, or external state management for this scenario?

Any guidance or architectural recommendations would be greatly appreciated.

Thank you very much.

[halturin]

Short answer first: the framework treats timers as local by design, and the duplicate you see is not really a timer problem. Below are the facts you can rely on, then brief sketches of the available solutions, because which one fits depends on numbers only you have.

Timers. SendAfter and SendEvery live inside the process that created them: not persisted, not migrated, not deduplicated. Node cron is the same one level up: stateless, node-scoped, and a job name is unique within the node, so the same job on every pod fires once per pod. That answers your second question. Local only, never responsible for reliable execution.

The cause. Two timers fire because two live activations of the same logical actor exist at the same moment. Process names are node-scoped (gen.ProcessID{Name, Node}), so nothing stops player_42 from existing on the old pod and the new one at once. Single activation is something you arrange; the framework does not hand it to you.

The invariant worth building around. Treat the actor as a write-through cache with an ownership claim and storage as the source of truth: due work lives in a row, every unit of work has an identity, writes are conditional. Then a second activation cannot produce a second reward, only a rejected write. No scheduler gives exactly-once by itself; at-least-once firing plus an idempotent handler is what also survives a crash between "timer fired" and "reward written".

Sketches: scheduling

Local timer, guarded handler. For work that only matters while the actor is alive (offline cleanup, periodic sync): keep the timer in the actor, re-derive it on activation from state you already persist (last_seen + 5m), and write the handler as a guarded transition ("if still offline since T, then clean up"). A duplicate run during a rollout is then harmless by construction. Zero infrastructure.

Durable due-work. For work that must happen whether or not anyone is around (rewards, delayed actions with side effects): a row with due_at and an identity such as (entity, job, scheduled_for) under a unique constraint. The timer becomes an optimisation for the common case; the row is the truth.

One runner per cluster, plus a sweep. For global schedules and for picking up overdue work nobody activated: elect a single runner with leader.Actor, add jobs in HandleBecomeLeader, remove them in HandleBecomeFollower. A periodic sweep over storage re-issues anything stranded by a lost message, a leader change or a dead pod. Have it wake owners rather than act for them, or it becomes a second writer.

Sketches: ownership

Registry-first. The entity claims a key when it activates and any node resolves the owner with a local read: grid (ships with 3.3.0) gives key-to-owner, lifecycle monitors and per-key groups. Simplest for client-driven activation. Read its docs before committing: it is AP, so a claim is not a lock, a losing owner is stopped by a refusable exit and must not flush stale state, and the shard count is fixed for the life of the domain.

Dispatcher-first. One leader-elected process decides where each entity activates; the target node's RegisterName is the local duplicate check (gen.ErrTaken means a contract violation, not a case to handle); a short suppression window on the dispatcher covers the gap before its view catches up; the sweep is the recovery path. No eventually-consistent ownership to reason about, at the cost of a central component on the activation path.

Sequenced handoff. When you want proof rather than a window: the new activation is told the current owner's PID, monitors it, then the old one is stopped through its own supervisor (a parent's exit cannot be refused, anyone else's can), and the new one starts working on the down message. A down with a real termination reason is proof the old owner flushed and its timers are gone; a down with gen.ErrNoConnection proves nothing, since it may be alive across a partition. Needs one authority per entity to mean anything.

Coarse-grained ownership. At large scale, do not own per player: own a room, a battle or a hash shard and keep players inside its owner. Fewer keys, co-location of players who talk to each other, and none of the per-login churn.

One hosting detail either way: give each entity instance its own restart Intensity with OnExceed: act.OnExceedDisable, or instances share the supervisor's global counter and one crash-looping entity takes down every entity under that supervisor. See Supervisor.

Sketch: the safety net

External lease. Where a duplicate is expensive, a lease in Redis, etcd or a Kubernetes Lease guards what the sketches above cannot: node failover and registration races. Worth it only if it hands you a monotonically increasing token that you carry into every write, so storage rejects a stale writer. A lock without one shrinks the window instead of closing it.

On the rollout, and what decides the rest

Stop being chosen first (readiness, route), then stop entities (cancel timers, flush state), and keep the grace period above your real drain time. But maxSurge > 0 deliberately overlaps pods and a pod can be unreachable while still alive, so choreography narrows the window and never closes it.

Which combination fits depends on: how many entities are live at once, hundreds or millions; the activation rate during a rollout; and whether your storage can do a conditional update. A couple of hundred players with Postgres wants the first two scheduling sketches plus registry-first ownership and nothing else. Millions with heavy intra-group messaging wants coarse-grained ownership, and probably a dispatcher, before anything else. Tell us the scale and the constraints and we can be specific.

Availability: leader.Actor is in the current release; grid arrives with 3.3.0 on September 4, 2026, with code on the v330 branches of ergo, application, actor and docs at devel.docs.ergo.services. The storage-side work is the same in every sketch, so nothing you build now is wasted.

[LIGUANGYAO]

Thanks for these great insights