RecordedOutput - Job Status Updates Throughout Job

[bmckay959]

We are doing some long running jobs (Some up to 2 hours), so we are looking to have some RecordedOutput that helps us see the state of the job and small updates to make sure everything is working as expected and we don't have a job that is just frozen.

What is the appetite for creating a 'hook' of some sort to allow the RecordedOutput to be persisted to the database throughout the job instead of only at the end?

Thanks

[brandur]

I could definitely see this being useful.

Similarly with riverlog. Currently, you'd have to wait for the entire job to finish before you'd see any logging output, and in the case of a long running job it'd be nice if you could tail the log output to see it streaming, even if it only updated once every 1 or 5 seconds or something.

I don't think implementation would be too difficult. It'd probably look like JobSetStateIfRunningMany's "metadata merge" section with a check that the job is still running.

API-wise, we'd probably want to add a new function like river.RecordOutputInterim to make it clear that this will be an extra DB call to get done.

@bgentry Thoughts?

[bgentry]

I think I'm mostly ok with it as a feature. The main hesitation I have with any feature that modifies job rows during execution is the potential for deadlocks or other bad interactions if transactions aren't closed properly or just if they're blocking system operations like completions, cancellation, etc from happening.

Use at your own risk, I suppose?

[bmckay959]

Yeah, so this is the concern for sure.

We also are only updating the data every 30 seconds, but the concern is still valid.....think of it more like a heartbeat more than anything else with very small amounts of data. For example, we log how many bytes has been processed to this point and loop iterations.

I think we could do something where we check to make sure the job is not completed before updating too, but we would need to wrap the logic in a lock of some sort, but I'm not 100% sure how postgres handles this and if we can put a row lock intentionally and what the repercussions of that would be. I believe below is roughly the right idea.

BEGIN;

-- Lock the row (skips if not found)
SELECT * FROM river_job
WHERE id = 123
FOR UPDATE;

-- Now check and update (this assumes you fetched the row above)
UPDATE river_job
SET metadata = 'data goes here'
WHERE id = 123
AND state != 'completed';

COMMIT;

[bmckay959]

Hey guys,

Just coming back to this. Is it worth me putting together a PR for this? Is there an appetite for this for others?

[brandur]

Took a rough stab at what this might look like in #1098.