Skip to content

Commit 8b21a75

Browse files
simusimu
committed
Add design document for "Commodore Component Dependencies"
1 parent 0cd609b commit 8b21a75

2 files changed

Lines changed: 198 additions & 0 deletions

File tree

‎docs/modules/SDDs/pages/0034-commodore-component-dependencies.adoc‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
= SDD 0034 - Commodore Component Dependencies
2+
3+
:sdd_author: Simon Gerber
4+
:sdd_owner: Aldebaran
5+
:sdd_reviewers: VSHN Tech Teams
6+
:sdd_date: 2026-08-17
7+
:sdd_status: draft
8+
include::partial$meta-info-table.adoc[]
9+
10+
[NOTE]
11+
.Summary
12+
====
13+
This SDD describes possible architectures which allow Commodore components to explicitly specify other components on which they depend.
14+
====
15+
16+
== Motivation
17+
18+
Currently, each Commodore component intrinsically depends on https://github.com/projectsyn/component-argocd[component-argocd].
19+
This dependency is special; Commodore https://github.com/projectsyn/commodore/blob/v1.34.1/commodore/component/compile.py#L266-L277[automatically injects a dummy `argocd.libjsonnet`] when compiling a component in isolation.
20+
21+
Additionally, many components depend on monitoring stack components (https://github.com/projectsyn/component-prometheus[component-prometheus], https://github.com/appuio/component-openshift4-monitoring[component-openshift4-monitoring]) or on https://github.com/projectsyn/component-espejote[component-espejote].
22+
23+
These dependencies can't explicitly be expressed at the moment.
24+
This has multiple drawbacks:
25+
26+
* Authors of components which depend on other components need to use workarounds to make component compilation work.
27+
The usual workarounds are fetching component libraries via `kapitan.dependencies` and replicating parts of the component's defaults.
28+
29+
* Commodore has no way to verify that a cluster catalog includes all the components on which other components depend.
30+
The catalog compilation only fails once a component library on which another component depends is missing.
31+
Some components provide better error messages by explicitly checking that their dependencies are present in the Reclass inventory's `applications` array.
32+
33+
We've been considering explicit component dependencies https://github.com/projectsyn/commodore/issues/690[for a long time] already.
34+
35+
However, recently there's been an uptick of needless busywork associated with component dependencies, because we've moved some Jsonnet logic from the monitoring component libraries to the shared https://github.com/projectsyn/jsonnet-libs[Jsonnet library repository].
36+
This code move requires that all components that want to use the monitoring component helper libraries need to explicitly depend on the Jsonnet library repository in order for `commodore component compile` to work.
37+
We would like to remove this explicit dependency on the Jsonnet library repository from components which only indirectly use the library repository through another component's component library.
38+
39+
Explicit component dependencies allow us to remove the explicit dependency on the Jsonnet library repository from most components.
40+
With explicit component dependencies, components don't need to know implementation details of component libraries that they consume.
41+
Instead, components can simply declare that they depend on the components whose libraries they use.
42+
This gives Commodore enough information to automatically include any direct component dependencies (including their Jsonnet dependencies) when compiling a component.
43+
44+
=== Goals
45+
46+
* Commodore components can specify other components on which they depend
47+
* Commodore uses component dependency information for component compilation
48+
* Commodore uses component dependency information for cluster catalog compilation
49+
50+
=== Non-Goals
51+
52+
* Automatic component inclusion in cluster catalogs based on dependency information
53+
* Automatic component version resolution
54+
* Nested component dependency resolution and fetching for component compilation
55+
56+
== Design Proposal
57+
58+
=== Dependency specification in the component
59+
60+
// TODO(sg): Dependency groups? (group all monitoring dependencies -> enables validation of non-overlapping CEL?)
61+
62+
// TODO(sg): use `<component_name>._metadata.dependencies` instead of `commodore.dependencies`? Would probably make parsing in catalog compile easier.
63+
64+
Commodore components can define other components on which they depend in Reclass parameter `commodore.dependencies`.
65+
Since the parameter is specific to the component (rather than part of the component's default configuration), it must be set in the component's `<component-name>.yml`, the same as `kapitan.compile`, `kapitan.dependencies` and `commodore.postprocess`.
66+
Configuring dependencies in `<component-name>.yml` ensures that the dependency definitions don't leak to other components.
67+
68+
The content of parameter `commodore.dependencies` is expected to be an object where each key is a component name.
69+
Commodore expects that the keys in `commodore.dependencies` follow the same rules as keys of https://syn.tools/commodore/reference/architecture.html#_dependency_discovery_and_versions[parameter `components`].
70+
Each entry of parameter `commodore.dependencies` is expected to hold an object with the following keys:
71+
72+
* `url`: A URL pointing to the component's Git repository.
73+
This field is used by component compilation to automatically fetch component dependencies.
74+
This field is ignored by catalog compilation.
75+
* `minversion`: An optional field which authors can use to specify a minimum required version for the dependency.
76+
Commodore parses this field as a https://semver.org/[semantic versioning ("SemVer")] version prefixed with a `v`.
77+
If the field isn't set, Commodore assumes that the component works with any version of the dependency.
78+
Notably, arbitrary https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-tree-ishalsotreeish[Git tree-ish]s (which are supported for parameter `components.<component-name>.version`) aren't supported in this field.
79+
* `requiredif`: An optional field which enables authors to express optional and situational dependencies (for example OpenShift-only, non-OpenShift-only).
80+
The value is expected to be a https://cel.dev/[Common Expression Language (CEL)] expression.
81+
The expression can reference cluster facts as `facts.<fact>`.
82+
Additionally the expression can reference component parameters (the rendered contents of `parameters.<component_name>`) as `config.<field>`.
83+
Other cluster metadata (dynamic facts, etc.) isn't available in the CEL expression.
84+
For dependencies which are completely optional, the field can simply be set to `false`.
85+
86+
==== Example
87+
88+
The following configuration models the dependencies of https://github.com/projectsyn/component-cilium[component-cilium] in version `v4.1.11`:
89+
90+
[source,yaml]
91+
----
92+
parameters:
93+
commodore:
94+
dependencies:
95+
openshift4-monitoring:
96+
url: https://github.com/appuio/component-openshift4-monitoring
97+
minversion: v6.11.3
98+
requiredif: 'facts.distribution == "openshift4" || facts.distribution == "oke"'
99+
prometheus:
100+
url: https://github.com/projectsyn/component-prometheus
101+
requiredif: '!(facts.distribution == "openshift4" || facts.distribution == "oke")'
102+
espejote:
103+
url: https://github.com/projectsyn/component-espejote
104+
requiredif: '(config.egress_gateway.enabled && config.egress_gateway.self_service_namespace.ips) || ((facts.distribution == "openshift4" || facts.distribution == "oke") && config.cilium_helm_values.kubeProxyReplacement == "strict")'
105+
openshift-nmstate:
106+
url: https://github.com/projectsyn/component-espejote
107+
requiredif: 'config.egress_gateway.enabled && config.egress_gateway.egress_ip_ranges.exists(r, has(config.egress_gateway.egress_ip_ranges[r].auto_egress_interfaces) && config.egress_gateway.egress_ip_ranges[r].auto_egress_interfaces)'
108+
----
109+
110+
=== Commodore component compilation
111+
112+
Commodore component compilation renders an initial inventory for the component with just the component definition and the user-supplied input.
113+
Commodore then reads `commodore.dependencies` from this initial inventory and determines which component dependencies to include based on each entry's `requiredif`.
114+
The caller can steer dependency fetching by setting suitable cluster facts and component configurations in the user-supplied input.
115+
116+
For dependencies which specify `minversion`, that version is used when compiling the component.
117+
Using the version specified in `minversion` (if any) for component compilation trivially ensures that component changes that require a newer dependency version than the one specified in `minversion` result in a compilation error.
118+
This ensures that component authors update their dependency `minversion` when they start using features introduced by a later version.
119+
120+
For all other dependencies, the dependency's default branch is used when compiling the component.
121+
Notably, using the oldest available tagged version of dependencies without `minversion` would be more equivalent to how we handle dependencies with explicit `minversion`.
122+
However, doing so would introduce a lot of overhead, and isn't necessary for the currently specified semantics of `minversion`.
123+
124+
After fetching the dependencies (using the already existing dependency fetching logic used in catalog compilation), the component compilation regenerates the temporary inventory with each dependency's defaults and component libraries included.
125+
Additionally, this new temporary inventory will have each dependency added to the `applications` array.
126+
127+
Afterwards, component compilation proceeds by calling Jsonnet-Bundler and Kapitan to render the component's manifests.
128+
129+
Notably, the implementation doesn't recursively fetch dependencies.
130+
Therefore, if any of a component's dependencies has further component dependencies, component authors must explicitly define those dependencies in the component compilation input files.
131+
132+
TIP: When using the Project Syn component template, we recommend adding any indirect component dependencies in the component's test cases.
133+
134+
=== Commodore catalog compilation
135+
136+
Commodore catalog compilation collects the component dependency specifications for all included components after fetching the components included by the catalog.
137+
The dependency specifications are unified by component name, and the minimum required version is the maximum of all specified minimum required versions.
138+
139+
As noted in section "<<_dependency_specification_in_the_component>>", Commodore expects that matching keys in parameters `components` and `<component_name>.commodore.dependencies` refer to the same component.
140+
This assumption significantly simplifies matching required dependency versions to components that are included in a catalog.
141+
142+
After dependency specifications are unified, Commodore verifies that each component dependency is explicitly included in the catalog with a compatible version
143+
Notably, verification treats any component version Git tree-ish which doesn't parse as a SemVer version (prefixed with `v`) as higher than all SemVer versions.
144+
Once that verification is completed, catalog compilation continues as usual.
145+
146+
Notably, catalog compilation won't try to automatically include component dependencies or adjust component versions.
147+
Making sure that a catalog includes all required components in a suitable version remains the responsibility of the operator defining the cluster catalog.
148+
This ensures that we can keep the single pass component fetching stage for catalog compilation.
149+
150+
=== Implementation Details/Notes/Constraints
151+
152+
As noted in section "<<_dependency_specification_in_the_component>>", the implementation assumes that matching keys in parameters `components` and `commodore.dependencies` refer to the same component.
153+
154+
We'll need to decide on a Python library which implements https://cel.dev[CEL].
155+
A quick search has found the following options:
156+
157+
* https://github.com/cel-expr/cel-python[Google's cel-expr/cel-python] which wraps the Google C++ implementation of CEL in a Python module.
158+
* https://github.com/hardbyte/python-common-expression-language[hardbyte/python-common-expression-language] which wraps the Rust https://crates.io/crates/cel[cel] crate in a Python module via https://pyo3.rs[PyO3].
159+
* https://github.com/cloud-custodian/cel-python[cloud-custodian/cel-python] which is a pure Python implementation of CEL.
160+
161+
We haven't decided on one of these libraries yet, but most likely we'll either use `cel-expr/cel-python` or `hardbyte/python-common-expression-language`.
162+
163+
For component version parsing, we'll probably use https://github.com/python-semver/python-semver[python-semver/python-semver].
164+
165+
=== Risks and Mitigations
166+
167+
When enabling component authors to define component dependencies with version constraints, there's an inherent risk that some components can't be included in the same cluster catalog due to dependency version conflicts.
168+
We don't expect this to be a problem as long as component authors use dependency version constraints sparingly.
169+
Additionally, since the proposal only allows specifying a minimum version, the risk is reduced already compared to an implementation that supports full version constraints.
170+
171+
//TODO(sg): add more if we find them during impl
172+
173+
== Drawbacks
174+
175+
The proposed implementation makes component compilation more complex.
176+
In particular, component compilation will consume more network traffic and disk space because we now download the full component dependency instead of just the library.
177+
Additionally, component compilation make take longer, both because we download more data and also because Commodore needs to do more setup work before the component can be compiled.
178+
179+
== Alternatives
180+
181+
=== Use Jsonnet-Bundler to implement component dependencies
182+
183+
Instead of extending Commodore itself to support explicit component dependencies, we could try to use https://github.com/projectsyn/jsonnet-bundler[Jsonnet-Bundler] to manage component dependencies.
184+
However, this approach has more drawbacks than the proposed design:
185+
186+
* Jsonnet-Bundler doesn't have a native way to express a minimum required version
187+
* Commodore catalog compilation would need a significant redesign to accommodate this approach
188+
** Commodore's component version management would need to be integrated with Jsonnet-Bundler
189+
** Commodore's component fetching would need to be replaced by Jsonnet-Bundler
190+
* Jsonnet-Bundler doesn't work well when multiple components depend on the same component but specify different versions.
191+
192+
== References
193+
194+
* https://github.com/projectsyn/commodore/issues/690[Commodore #690: Add support for explicit component dependencies]
195+
* https://github.com/projectsyn/jsonnet-bundler[projectsyn/jsonnet-bundler]
196+
* https://cel.dev[Common Expression Language (CEL)]
197+
* https://semver.org[Semantic Versioning ("SemVer")]

‎docs/modules/SDDs/pages/index.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,4 @@ The all are using the xref:sdd-template.adoc[SDD Template].
2828
* xref:0031-component-version-tracking.adoc[0031 - Central Component Version tracking]
2929
* xref:0032-compile-pipeline.adoc[0032 - Commodore Compile Pipeline]
3030
* xref:0033-commodore-component-instance-versioning.adoc[0033 - Commodore Component Instance Versioning]
31+
* xref:0034-commodore-component-dependencies.adoc[0034 - Commodore Component Dependencies]

0 commit comments

Comments
 (0)