New channel for in-progress development distribution

[martinkrulltott]

Currently Analytics apps use alpha versions in the development process to facilitate testing and sharing code, context switching and pull request reviews.
Manually publishing public, permanent, immutable package versions for our in-progress development distribution is not the right approach. We need another channel to facilitate this.
Based on an internal discussion on Slack.

[varl]

@martinkrulltott could you provide a summary of the discussion for context? I cannot access the thread you linked.

[varl]

Publishing to "alpha" does have some drawbacks, but before assuming what the context is for this discussion I have questions.

To clarify some terminology, with "channel" do you mean:

• an NPM "dist-tag"?
• NPM as a distribution channel?

Manually publishing public, permanent, immutable package versions for our in-progress development distribution is not the right approach.

Why is developing in the open and publicly releasing immutable development builds of our libraries not the right approach?

[martinkrulltott]

Hey @varl , sorry for the internal Slack link and lack of backstory/info.
I was discussing this with @amcgee last week, but as he was going on vacation he asked me to create a note for this issue.
The description above is actually Austin's words / opinions, not mine.
It came as a response to me publishing alpha versions of Analytics manually to facilitate development, testing and sharing between devs, as our setup for using a local Analytics branch in our apps (e.g. Data Visualizer) isn't ideal and has a few pitfalls which can prevent consistent output.
For me lately, working on multiple tickets simultaneously which rely on Analytics branches, using manually published alpha versions has been a real life-saver.
The process has looked like this:

1. Implement a feature / bug fix on a Analytics branch

2. Once the development is done and it's ready to be implemented in an app (usually DV), publish a manual alpha version, this becomes sort of a checkpoint for the development.

3. In the app, use the not-yet-released Analytics changes by bumping the Analytics dep to the new alpha version

4. Once everything is implemented, get the Analytics change reviewed and released + update the app dep to the newly released version (Instead of alpha)

If the need for an Analytics change is needed during the app development, go back to step 2-3, using a new alpha version.

Examples of Analytics branches / alphas used in this way are:
dhis2/analytics#527, dhis2/analytics#508, dhis2/analytics#529

Using the alpha versions as checkpoints means that app development / testing / reviewing / pair-programming can be performed without our "special" manual way of setting up local Analytics <-> local app and makes switching back and forth between different ones a lot easier. The app branch can thus be used independently, without any knowledge of how to setup Analytics and it completely removes the risk of running in to issues caused by mismatching Analytics versions.

The only drawback I've found so far is the fact that manually publishing alpha versions means you manually have to select the version number to publish (e.g. dhis2/analytics#529 uses 8.3.0-alpha), which could obviously become an issue if you choose the wrong one or if they're not subsequently released in the order they were created. However, with only 1-2 devs simultaneously creating alphas I don't think this is very hard to coordinate.

But @amcgee didn't agree and prefer a different solution, which I'm not yet fully across. But I hope this note will be a good starting point for a discussion 👍

[amcgee]

@varl the main thing I have issue with here is that these aren’t true alpha releases (they’re never on the alpha branch). Currently, a dev manually runs npm publish to push the pre-release tagged version of a feature branch. The issue here is that there’s no record of the code used for a particular tag on npm, it’s dangerous because a dev could accidentally publish a production release manually (we want to move away from having individual npm publish credentials), and the npm package gets crowded with lots of non-production bundles.

I fully agree we should keep things open, immutable, and permanent but manually publishing to npm isn’t the right way to do that (publishing to GitHub packages from a github action might be)

[varl]

The only drawback I've found so far is the fact that manually publishing alpha versions means you manually have to select the version number to publish (e.g. dhis2/analytics#529 uses 8.3.0-alpha), which could obviously become an issue if you choose the wrong one or if they're not subsequently released in the order they were created. However, with only 1-2 devs simultaneously creating alphas I don't think this is very hard to coordinate.

I am reasonably confident that semantic-release figures out what version of alpha to publish regardless of what package.json says is the version: https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/pre-releases.md, so there should be no need for manual coordination on *-alpha.{version} strings.

[amcgee]

@varl these “alpha” releases do not use the alpha branch, which works as you would expect. They are manually published to npm by a dev on a local machine. That’s what I’m saying we need to avoid.

alpha works well but only AFTER a feature branch is “ready” (not before when it is still a work in progress) and also doesn’t work super well for backports - Analytics has version branches for 2.33 and 2.32 which might receive updates but aren’t the head versions

[varl]

Thanks for the context, @martinkrulltott & @amcgee ! :)

As far as immutable/permanent/open for devleopment builds goes, we use d2-ci to accomplish that for apps, and for a while we used d2-ci to store library artifacts as well -- although pulling from Git using NPM is a rough edge in development scenarios so not the smooth experience we would like.

Using GitHub Packages for development could fill that gap. The --publish argument can be extended to support github easily.

[varl]

@varl these “alpha” releases do not use the alpha branch, which works as you would expect. They are manually published to npm by a dev on a local machine. That’s what I’m saying we need to avoid.

alpha works well but only AFTER a feature branch is “ready” (not before when it is still a work in progress) and also doesn’t work super well for backports - Analytics has version branches for 2.33 and 2.32 which might receive updates but aren’t the head versions

Gotcha. Thanks for clarifying, I did assume "manual" meant "manually raising and merging a PR to alpha branch".

e: While I do think it is fine to merge a feature to alpha that is half-baked and iterate on it, and that it may be cut before release if we realize that we do not need it anymore, what we need here is slightly different from an alpha in that regard.