CPAN installation as a test, with GitHub workflow

If I made a distribution Foo and uploaded it to CPAN, I'd expect that it is install-able via various CPAN clients such as cpan, cpanm, cpm, that is, from a fresh perl installation, these commands should be successful:

cpan Foo
cpanm Foo
cpm install -g Foo

What's a bit non-obvious is that this requires a chain of conventions to be satisfied in order to happen smoothly... distribution Foo needs to properly ship with build procedure / installer and META.json, in which all dependencies are correctly declared.

But that is not enough, even if I correctly authored all the meta-data in distribution Foo, if there is even a single miss in any other dependencies in Foo's dependency tree, the installation would fail.

For that reason, I thought t might be worth it to test the installation process during development, at least make a CI workflow that tries to install everything. The successful of such test would just be whether the installation is successfully finished or not. Conventionally the exit status ($?) reflects that.

Here's something I came up with. It is a GitHub workflow that first build a distribution, a .tar.gz file, then try to install that file with cpanm:

name: Installation Test

on:
  push:
    branches:
      - fun

jobs:
  build:
    runs-on: 'ubuntu-latest'
    steps:
      - uses: actions/checkout@v2
      - name: Setup Perl environment
        uses: shogo82148/actions-setup-perl@v1
      - name: Install Authoring Dependencies
        run: cpanm -n App::ModuleBuildTiny && cpanm -q -n --no-man-pages --installdeps .
      - name: Generate dist
        run: mbtiny dist
      - name: Display what is generated
        run: |-
          echo *.tar.gz
          tar tvzf *.tar.gz
      - uses: actions/upload-artifact@v2
        with:
          name: dist-for-installation-test
          path: '*.tar.gz'
          retention-days: 1
  install:
    runs-on: 'ubuntu-latest'
    needs: build
    container:
      image: perl:5.34
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: dist-for-installation-test
      - name: Display the downloaded files
        run: ls -R
      - name: Install in a perl:5.34 container (--notest)
        run: cpanm --notest *.tar.gz

There are a lot of instructions but what matters is the final one cpanm --notest *.tar.gz. This would install whatever was built and all its dependencies, without running module tests`, and if even that failed, that means some meta data is missing and the distribution would not be install-able even if it is uploaded to CPAN.

Of course that is just cpanm, I could also add cpan and cpm there to verify whether there is a difference between all these CPAN clients. Also, I could setup a matrix run so it runs on all perl versions.

This workflow is currently used in Perl-Critic-TooMuchCode, with a few runs already finished here.

The workflow contains two jobs, build and install. install needs build so it is always executed after successful build.

The build job runs on the default ubuntu-latest machine, but the install job runs in a container instance with presumably a different version of perl, and also without a clone of current repo. With this setup I can simulate the installation process on a fresh machine.

To share the distribution .tar.gz file across different jobs, the only way I can fin is by uploading the file to the "artifact" storage -- basically an external storage. This probably cost me something if a lot of files are accumulated. I changed the retention period to 1 day because I don't plan to download this anyway.

On the other hand, it could be part of doing the actual CPAN release. Maybe with some modification the workflow would build a new version with new version number, and I would just have to download the artifact then re-upload to CPAN. This would save the setup of preparing authoring tools. It could be a useful scenario for teaching new developers to upload something to CPAN.

The installation process also download a lot of stuffs from cpan.org and cost some bandwidth on the way. It's best not to do so on every commits or on a branch with frequent pushes. Definitely suitable pre-release though.

Most likely, this idea of "Installation as a test" isn't new, and it is a bit convenient to have it checked in CI.