What is your environment(Kubernetes version, Fluid version, etc.)
- Kubernetes v1.30.0 (kind v0.23.0), 2 worker nodes
- Fluid built from master
- Runtime: CacheRuntime
Describe the bug
CacheEngine.getRuntimeInfo() builds its RuntimeInfo with an empty tiered store instead of the one declared in the spec:
// pkg/ddc/cache/engine/runtime.go:100
base.WithTieredStore(datav1alpha1.TieredStore{}),
Every other engine passes the real value — pkg/ddc/alluxio/runtime_info.go:33, and likewise juicefs, jindo, jindofsx, jindocache, efc, thin and vineyard all pass runtime.Spec.TieredStore. CacheRuntime is the only one passing a zero struct.
That empty value propagates all the way to the node labels:
convertToTieredstoreInfo returns an empty TieredStoreInfo as soon as len(Levels) == 0 (pkg/ddc/base/runtime.go:465).
tieredstore.GetLevelStorageMap iterates over GetTieredStoreInfo().Levels and therefore returns an empty map (pkg/utils/tieredstore/tiered_store.go:60-83).
labelNodeWithCapacityInfo ranges over that empty map (pkg/utils/dataset/lifecycle/node.go:330-342), so fluid.io/s-h-cache-m-<ns>-<name> and fluid.io/s-h-cache-d-<ns>-<name> are never added at all, while totalRequirement keeps its initial resource.MustParse("0Gi") and fluid.io/s-h-cache-t-<ns>-<name> is written unconditionally as 0B.
So a CacheRuntime that declares a real tiered store still advertises zero cache capacity on every node it lands on.
What you expect to happen:
The capacity labels should reflect spec.worker.tieredStore: the total label carrying the summed quota, and the memory / disk labels present according to the media actually configured — the same behaviour the other runtimes already have.
How to reproduce it
- Create a CacheRuntime whose worker declares a non-zero tiered store:
spec:
worker:
replicas: 2
tieredStore:
levels:
- emptyDir: {quota: 1Gi}
high: "0.8"
low: "0.5"
- Wait for the worker pods to be scheduled, then read the labels off a node running one:
kubectl get node <node> -o json | jq '.metadata.labels | with_entries(select(.key | contains("fluid.io")))'
Observed on both worker nodes:
fluid.io/dataset-num = 1
fluid.io/s-cache-default-mooncake-demo = true
fluid.io/s-default-mooncake-demo = true
fluid.io/s-h-cache-t-default-mooncake-demo = 0B <-- 1Gi was configured
There is no fluid.io/s-h-cache-m-... and no fluid.io/s-h-cache-d-... label at all — the loop that would
add them never runs.
Worth contrasting: the same quota is read correctly everywhere else. The bound Dataset reports the expected
total for 2 replicas of 1Gi:
NAME UFS TOTAL SIZE CACHED CACHE CAPACITY CACHED PERCENTAGE PHASE
mooncake-demo 2.00GiB 0B 2.00GiB 0.0 ...
So the tiered store spec itself is fine; only the path that feeds the node labels sees an empty value.
Additional Information
A fix needs a conversion from RuntimeTieredStore to the legacy TieredStore that base.WithTieredStore consumes. Two things are worth agreeing on before implementing, because they are product decisions rather than mechanical ones:
-
Medium mapping. RuntimeTieredStoreLevel (api/v1alpha1/cacheruntime_types.go:275) has no MediumType field. It describes the medium structurally instead, through ProcessMemory, EmptyDir and HostPath, whereas the legacy Level (api/v1alpha1/common.go:39) carries an explicit MEM/SSD/HDD enum. ProcessMemory and an EmptyDir with medium: Memory clearly map to MEM. HostPath has no way to say whether it is backed by SSD or HDD, and GetLevelStorageMap splits precisely on MEM versus SSD|HDD, so this needs a decision — default HostPath to one of them, or extend the API.
-
Whether the client tier counts. tieredStore appears twice in the CacheRuntime spec: on the worker (cacheruntime_types.go:114) and on the client (cacheruntime_types.go:123). These labels feed worker placement during scheduling, so it should be settled explicitly whether client-side cache contributes to a node's advertised capacity.
Note this is a separate problem from the tiered store memory quota being dropped from the worker memory limit (#6166, fixed by #6167): that one concerns the container's memory accounting, this one concerns the node labels. They share the tieredStore spec field but neither fix depends on the other.
What is your environment(Kubernetes version, Fluid version, etc.)
Describe the bug
CacheEngine.getRuntimeInfo()builds itsRuntimeInfowith an empty tiered store instead of the one declared in the spec:Every other engine passes the real value —
pkg/ddc/alluxio/runtime_info.go:33, and likewise juicefs, jindo, jindofsx, jindocache, efc, thin and vineyard all passruntime.Spec.TieredStore. CacheRuntime is the only one passing a zero struct.That empty value propagates all the way to the node labels:
convertToTieredstoreInforeturns an emptyTieredStoreInfoas soon aslen(Levels) == 0(pkg/ddc/base/runtime.go:465).tieredstore.GetLevelStorageMapiterates overGetTieredStoreInfo().Levelsand therefore returns an empty map (pkg/utils/tieredstore/tiered_store.go:60-83).labelNodeWithCapacityInforanges over that empty map (pkg/utils/dataset/lifecycle/node.go:330-342), sofluid.io/s-h-cache-m-<ns>-<name>andfluid.io/s-h-cache-d-<ns>-<name>are never added at all, whiletotalRequirementkeeps its initialresource.MustParse("0Gi")andfluid.io/s-h-cache-t-<ns>-<name>is written unconditionally as0B.So a CacheRuntime that declares a real tiered store still advertises zero cache capacity on every node it lands on.
What you expect to happen:
The capacity labels should reflect
spec.worker.tieredStore: the total label carrying the summed quota, and the memory / disk labels present according to the media actually configured — the same behaviour the other runtimes already have.How to reproduce it
Observed on both worker nodes:
There is no
fluid.io/s-h-cache-m-...and nofluid.io/s-h-cache-d-...label at all — the loop that wouldadd them never runs.
Worth contrasting: the same quota is read correctly everywhere else. The bound Dataset reports the expected
total for 2 replicas of 1Gi:
So the tiered store spec itself is fine; only the path that feeds the node labels sees an empty value.
Additional Information
A fix needs a conversion from
RuntimeTieredStoreto the legacyTieredStorethatbase.WithTieredStoreconsumes. Two things are worth agreeing on before implementing, because they are product decisions rather than mechanical ones:Medium mapping.
RuntimeTieredStoreLevel(api/v1alpha1/cacheruntime_types.go:275) has noMediumTypefield. It describes the medium structurally instead, throughProcessMemory,EmptyDirandHostPath, whereas the legacyLevel(api/v1alpha1/common.go:39) carries an explicitMEM/SSD/HDDenum.ProcessMemoryand anEmptyDirwithmedium: Memoryclearly map toMEM.HostPathhas no way to say whether it is backed by SSD or HDD, andGetLevelStorageMapsplits precisely onMEMversusSSD|HDD, so this needs a decision — defaultHostPathto one of them, or extend the API.Whether the client tier counts.
tieredStoreappears twice in the CacheRuntime spec: on the worker (cacheruntime_types.go:114) and on the client (cacheruntime_types.go:123). These labels feed worker placement during scheduling, so it should be settled explicitly whether client-side cache contributes to a node's advertised capacity.Note this is a separate problem from the tiered store memory quota being dropped from the worker memory limit (#6166, fixed by #6167): that one concerns the container's memory accounting, this one concerns the node labels. They share the
tieredStorespec field but neither fix depends on the other.