add spack retry to CI

[rfbgo]

This will help us be more resilient to GH flakiness by being able to retry:

Step #2 - "ramble-unit-tests": us-central1-docker.pkg.dev/ramble-eng/ramble-repo/ramble-rockylinux-8-spackv1.0.0-python3.7.17:latest
Step #2 - "ramble-unit-tests": branch 'develop' set up to track 'origin/develop'.
Step #2 - "ramble-unit-tests": fatal: unable to access 'https://github.com/spack/spack-packages.git/': Empty reply from server
Step #2 - "ramble-unit-tests": ==> Warning: Error constructing repository '/builder/home/.spack/package_repos/fncqgg4': Failed to clone repository builtin

It also DRYs the env setup into a shared script to

[ramble-project-pr-bot]

Ramble Performance Test Metrics

Results produced with commit: 86a44be

Test Name	Outcome	Duration (s)	Most Recent Run (s)	Last 5 Avg (s)
test_analyze_large_file	passed	1.0556	1.0590 (b7a264f)	1.0648
test_large_template_expansion	passed	1.1993	1.1979 (b7a264f)	1.3086
test_many_experiments	passed	24.1654	23.4773 (b7a264f)	24.0185
test_many_objects_defaults	passed	12.5638	12.3728 (b7a264f)	12.4969
test_matrix_filter_perf	passed	1.0854	1.0663 (b7a264f)	1.1014

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[linsword13]

A bit outside the scope, but I wonder if this set -e would defeat the retry somehow?

[rfbgo]

That is a very good question. My first reaction when I saw this failure was "oh I'm surprised we don't have set -e? It should have saved me?". I think you're right here and set -e will defeat my intention in this specific yaml

I guess there is an implied choice here between "fail fast" (-e) and "retry and fail clearly". Any preference?