Add Ymir staging environment strategies research by majamassarini · Pull Request #236 · packit/research

majamassarini · 2026-07-09T14:38:36Z

Documents two complementary approaches for Ymir pre-production validation:

Full Staging Environment - Persistent production-like deployment with real external service integration for release candidate testing
CI with Testing Farm - Automated workflow validation on every PR using ephemeral OpenShift Local clusters on internal Red Hat Ranch

Assisted-by: Claude Sonnet 4.5 noreply@anthropic.com

Documents two complementary approaches for Ymir pre-production validation: 1. **Full Staging Environment** - Persistent production-like deployment with real external service integration for release candidate testing 2. **CI with Testing Farm** - Automated workflow validation on every PR using ephemeral OpenShift Local clusters on internal Red Hat Ranch Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>

lbarcziova

nice write-up!

lbarcziova · 2026-07-21T08:45:18Z

+
+- Dedicated `ymir-staging` bot credentials
+- Staging Jira instance (`stage-redhat.atlassian.net`)
+- Agents run with `STAGE_INSTANCE=true` environment variable - processes issues fully, creates branches in dist-git, adds Jira comments, but skips MR creation


could you elaborate more about the strategy regarding dist-git? Would this be using production repos or having some dedicated namespace? Considering the part about creation of branches, which has impact, we should do this in separate namespace/place.

skips MR creation

Any particular reason for this? I would see a benefit in experiencing the full flow, e.g. when changing anything related to the MR creation implementation

I suppose the MRs are created exclusively against the package itself, rather than fork.

ymir works on its own fork, and if we have staging credentials, we will have a complete new fork.

However sometimes ymir creates branches on dist-git directly, and that is the only part that I see as "dangerous". But it is "dangerous" also when we do it via production, because the MR could be not-accepted and the new branch we created could be not needed...

We should pay attention to that, but I don't see it as blocking, and we should be able to revert it if needed.

afaik there is also staging dist-git (cgit) but I do wonder if there is a staging environment for the gitlab repos so we wouldn't create these Z-stream branches in production dist-git

lbarcziova · 2026-07-21T08:47:49Z

+
+### What It Is
+
+Complete Ymir deployment (11 Deployments, 2 CronJobs) on dedicated infrastructure with:


we could certainly go with more minimal deployment, skip redis-commander, most of the cronjob and maybe even more

lbarcziova · 2026-07-21T08:49:46Z

+4. Clone test issue from production Jira to staging Jira
+5. Agents process issue through full workflow (triage → backport → rebase → rebuild)


would this be always run on one issue or one per use-case, or something else?

lbarcziova · 2026-07-21T08:52:01Z

+
+### Recommended Strategy
+
+**Use both approaches as complementary layers:**


lbarcziova · 2026-07-21T08:59:20Z

+
+**Proof of Concept:** The `testing-farm-automation` branch demonstrates Testing Farm VM provisioning and Ymir deployment to CRC, but lacks validation agent, issue cloning integration, and internal Testing Farm testing.
+
+**Suggested Strategy:** Start with **CI with Testing Farm** for lower initial cost and automatic test feeding. Gather metrics (test success/failure rates, validation results) to evaluate if staging instance is needed for complementary validation coverage. Key unknown to verify first: dist-git access from internal Testing Farm Red Hat Ranch VMs.


I'm slightly inclined to flip the order and start with the staging instance - to me this directly addresses the "does this break prod" concern by validating the actual deployment against diverse real issues before promotion, which per-PR testing as a pre-merge gate for individual changes doesn't cover, at least with my current understanding of that approach, but happy to discuss this more.

lbarcziova · 2026-07-21T09:00:43Z

+
+**jira_prod_to_stage Cloner:**
+
+SE team has implemented a [jira_prod_to_stage cloner](https://gitlab.cee.redhat.com/abobrov/jotnar-se/-/tree/feature/jira-prod-to-stage-cloner) tool that:


nice! Would be great to reuse this

majamassarini · 2026-07-23T14:09:58Z

@lbarcziova thanks for the review! I will respond here in one comment and also I will try to explain what my concerns are about focusing only on the staging instance.

I'm slightly inclined to flip the order and start with the staging instance

Yesterday when I deployed the phoenix+postgres feature, I really missed the staging instance and only a real staging instance would have helped me investigate the issues I had beforehand. So I am not against it.
However, I think that we make openshift changes, that can be tested only in a staging instance, quite rarely. And what I miss the most is something that tells us whether the changes in our prompt/tools in our PRs are good enough.
I know we already have the e2e tests but they aren't integration tests, they are limited in the sense that they don't interact with the tools and also they run in dry run mode, so they skip a lot of code.

In packit we use the staging instance to perform "integration tests"; we release our repos and we react to all the events there.
I can't think of anything similar in ymir. We need to create some ad-hoc tests to automatically feed the staging instance and evaluate its behavior, and when you have a test for it, in my mind, the most visible place for a test is the CI.
Adding a full openshift cluster in a testing farm machine is probably an overkill, so we could also go with podman for the integration test; and @mfocko already did much of the work for it. Even more, if we will also have the real staging instance.

would this be always run on one issue or one per use-case, or something else?

All I could think of is to take one to three issues (depending on how much time we want to invest in the test) from our query and run them against the complete chain (and have an agent to review the work done). We may end up not really testing the changes in the PR if, for example, the changes are for a rebuild issue and we test them against a backport issue. But at least as soon as we process a rebuild our tests would fail - ideally before we do it in production... Otherwise, we would need an agent to pick up the best issue for the test, and this would increase the complexity.

Any particular reason for this? I would see a benefit in experiencing the full flow, e.g. when changing anything related to the MR creation implementation

We should create the MR, just not push it, we should save it for review by the test agent reviewer.
I fear that replacing the real gitlab repos would be quite complex. I am not completely against it but I am not sure if it is worth the effort and the drift we create with the real code. My point is, if we don't create the MR we won't create too much noise for the users. And here you have a point with the branch creation, which is the only real problem I see with working with the real gitlab instance. In my mind if the review agents tell us we created the branch when not needed - and we should definetely check for this - we can remove it, right? I hope it won't happen too often.

So to summarize it up.
We can go with a real staging instance first. But for that to be useful we would still need to write some sort of integration test that would run automatically, quite often, and we would need to access its result easily.

mfocko

Overall looks nice, I have left “some doubts behind” in the review :D

mfocko · 2026-07-23T15:20:08Z

+
+## Context
+
+Ymir needs pre-production validation to catch deployment issues, infrastructure problems, and integration failures before they hit production. Two complementary approaches address different validation needs:


infrastructure problems

this feels quite brave… tbf I’m not sure how I feel about this, because, if this had worked, it would’ve caught even issues, very likely, unrelated to the changes in the PR and choke the whole merging process…

Good point, when I was writing it, I had similar thoughts. I struggle finding a balance between good visibility, which CI could give us, and practice - don't choke the merging process. I was thinking that they could be not mandatory... but since they would take a lot of time, probably in the end, nobody will notice them...

mfocko · 2026-07-23T15:22:54Z

+Ymir needs pre-production validation to catch deployment issues, infrastructure problems, and integration failures before they hit production. Two complementary approaches address different validation needs:
+
+1. **Full Staging Environment** - persistent production-like deployment for integration testing with real external services
+2. **CI with Testing Farm** - automated deployment validation on each PR using temporary OpenShift Local clusters


I’d doubt the reliability for reproducing deployment issues… regardless of whether we’re talking about podman-compose deployment (far from production cluster), or even the DIY OpenShift setup as we have for Packit (it is still not the same thing, and very likely limited by the Testing Farm environment itself).

I agree, things like firewall rules or network issues, could pass uncaught.

I’ll pick on firewall specifically… I think we would need quite extensive test suite to catch that :/

mfocko · 2026-07-23T15:25:21Z

+
+- Dedicated `ymir-staging` bot credentials
+- Staging Jira instance (`stage-redhat.atlassian.net`)
+- Agents run with `STAGE_INSTANCE=true` environment variable - processes issues fully, creates branches in dist-git, adds Jira comments, but skips MR creation


I suppose the MRs are created exclusively against the package itself, rather than fork.

mfocko · 2026-07-23T15:26:42Z

+**Automated CI:**
+
+1. PR opened → Testing Farm provisions CentOS Stream 10 VM
+2. VM installs CRC, starts OpenShift cluster


I still don’t understand why TF doesn’t provide a ready-to-use image for this…

mfocko · 2026-07-23T15:29:57Z

+
+**Manual Testing (same infrastructure):**
+
+1. Developer runs `tmt run --interactive` to provision Testing Farm VM


doesn’t exist… the closest thing would be testing-farm request --reserve, though it’s still mostly: run the tests and keep the machine for SSH, if needed

yes, right. It was an hallucination that I didn't fix, after trying it several times without success 😅

mfocko · 2026-07-23T15:33:12Z

+- ✅ **Reusable for manual testing** - same infrastructure for CI and developer testing
+- ✅ **No local hardware needed** - Testing Farm provides VMs with sufficient resources
+- ❌ **Slower feedback** - VM provisioning + CRC startup adds 5-10 minutes (can be improved with custom Testing Farm images, see [Build custom images for Testing Farm](https://fedoramagazine.org/build-custom-images-for-testing-farm/))
+- ❌ **Requires test automation** - needs validation agent creation to verify correctness


I think we should also consider the reliability of validation agent.

yes true, however I see it as needed in both solutions. I mean, the staging instance, without some sort of test running there automatically isn't very useful, from my point of view.

mfocko · 2026-07-23T15:33:32Z

+
+The same Testing Farm setup serves both automated CI and manual developer testing:
+
+**For CI:** PR opened → VM provisions → CRC starts → Ymir deploys → validation runs → results to PR → VM tears down


I would do that only for gating, this is quite expensive…

mfocko · 2026-07-23T15:34:35Z

+
+**Note on Credentials:**
+
+Both staging instance and CI can reuse **production Ymir credentials** initially, since they do not make any user-facing changes. This avoids credential management overhead and speeds up initial setup.


since they do not make any user-facing changes

well…

mfocko · 2026-07-23T15:37:43Z

+
+- Can't create test issues in production Jira
+- Can't open MRs in production dist-git
+- Can't reuse processed issues (agents detect "already fixed")


What are we going to do once the tests fail mid-run and leave the ticket in there? Is there going to be some kind of a clean up?

I don't think we need something, we worked on staging jira with a clone of an issue that will never been picked up again (ideally) by our tests, and the branches we create will be cleaned up by the existing code. So I don't see much else here.

mfocko · 2026-07-23T15:40:19Z

+
+1. User runs tmt with issue key parameter:
+   ```bash
+   tmt run --environment YMIR_TEST_ISSUE=RHEL-12345 plan --name ymir-manual-test/remote


That won’t work… and I’m not going to lie, but using tmt feels like launching a rocket into space, so… these commands will be pretty big :D

we can definitely try to mock something up, but it won’t be an easy thing to do

mfocko · 2026-07-23T15:47:16Z

However, I think that we make openshift changes, that can be tested only in a staging instance, quite rarely.

👍

Adding a full openshift cluster in a testing farm machine is probably an overkill, so we could also go with podman for the integration test;

the thing is that neither podman, nor OpenShift on Testing Farm will provide the same environment… Yes, you would “deploy” it on OpenShift, but you still have no guarantee that it will have the same outcome as the production cluster…

We should create the MR, just not push it, we should save it for review by the test agent reviewer.

wdym by that? 🤔 I think the best would be opening some labeled MRs against the fork, but… yeah, not sure how easy it is to adjust to that

majamassarini · 2026-07-24T09:19:26Z

wdym by that? 🤔 I think the best would be opening some labeled MRs against the fork, but… yeah, not sure how easy it is to adjust to that

we will have the changes in the fork, we can save the output that the agent would use for creating the MR description somewhere, and I think we would have everything necessary for reviewing the job done.

usercont-release-bot added this to Packit pull requests Jul 9, 2026

github-project-automation Bot moved this to New in Packit pull requests Jul 9, 2026

majamassarini force-pushed the ymir-staging-strategies branch from ac18eba to 3fda4f5 Compare July 10, 2026 09:44

majamassarini force-pushed the ymir-staging-strategies branch from 3fda4f5 to c3bfbe5 Compare July 10, 2026 09:52

lbarcziova reviewed Jul 21, 2026

View reviewed changes

mfocko reviewed Jul 23, 2026

View reviewed changes


		### What It Is

		Complete Ymir deployment (11 Deployments, 2 CronJobs) on dedicated infrastructure with:

		4. Clone test issue from production Jira to staging Jira
		5. Agents process issue through full workflow (triage → backport → rebase → rebuild)


		### Recommended Strategy

		Use both approaches as complementary layers:


		Proof of Concept: The `testing-farm-automation` branch demonstrates Testing Farm VM provisioning and Ymir deployment to CRC, but lacks validation agent, issue cloning integration, and internal Testing Farm testing.

		Suggested Strategy: Start with CI with Testing Farm for lower initial cost and automatic test feeding. Gather metrics (test success/failure rates, validation results) to evaluate if staging instance is needed for complementary validation coverage. Key unknown to verify first: dist-git access from internal Testing Farm Red Hat Ranch VMs.


		jira_prod_to_stage Cloner:

		SE team has implemented a [jira_prod_to_stage cloner](https://gitlab.cee.redhat.com/abobrov/jotnar-se/-/tree/feature/jira-prod-to-stage-cloner) tool that:


		## Context

		Ymir needs pre-production validation to catch deployment issues, infrastructure problems, and integration failures before they hit production. Two complementary approaches address different validation needs:


		Manual Testing (same infrastructure):

		1. Developer runs `tmt run --interactive` to provision Testing Farm VM


		The same Testing Farm setup serves both automated CI and manual developer testing:

		For CI: PR opened → VM provisions → CRC starts → Ymir deploys → validation runs → results to PR → VM tears down


		Note on Credentials:

		Both staging instance and CI can reuse production Ymir credentials initially, since they do not make any user-facing changes. This avoids credential management overhead and speeds up initial setup.

Uh oh!

Conversation

majamassarini commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lbarcziova left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majamassarini commented Jul 23, 2026

Uh oh!

mfocko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mfocko commented Jul 23, 2026

Uh oh!

majamassarini commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

majamassarini commented Jul 9, 2026 •

edited

Loading