Skip to content

[BUG]CacheRuntime: RuntimeInfo is built with an empty TieredStore, so cache capacity node labels are never populated #6174

Description

@btxu-db

What is your environment(Kubernetes version, Fluid version, etc.)

  • Kubernetes v1.30.0 (kind v0.23.0), 2 worker nodes
  • Fluid built from master
  • Runtime: CacheRuntime

Describe the bug

CacheEngine.getRuntimeInfo() builds its RuntimeInfo with an empty tiered store instead of the one declared in the spec:

// pkg/ddc/cache/engine/runtime.go:100
base.WithTieredStore(datav1alpha1.TieredStore{}),

Every other engine passes the real value — pkg/ddc/alluxio/runtime_info.go:33, and likewise juicefs, jindo, jindofsx, jindocache, efc, thin and vineyard all pass runtime.Spec.TieredStore. CacheRuntime is the only one passing a zero struct.

That empty value propagates all the way to the node labels:

  1. convertToTieredstoreInfo returns an empty TieredStoreInfo as soon as len(Levels) == 0 (pkg/ddc/base/runtime.go:465).
  2. tieredstore.GetLevelStorageMap iterates over GetTieredStoreInfo().Levels and therefore returns an empty map (pkg/utils/tieredstore/tiered_store.go:60-83).
  3. labelNodeWithCapacityInfo ranges over that empty map (pkg/utils/dataset/lifecycle/node.go:330-342), so fluid.io/s-h-cache-m-<ns>-<name> and fluid.io/s-h-cache-d-<ns>-<name> are never added at all, while totalRequirement keeps its initial resource.MustParse("0Gi") and fluid.io/s-h-cache-t-<ns>-<name> is written unconditionally as 0B.

So a CacheRuntime that declares a real tiered store still advertises zero cache capacity on every node it lands on.

What you expect to happen:

The capacity labels should reflect spec.worker.tieredStore: the total label carrying the summed quota, and the memory / disk labels present according to the media actually configured — the same behaviour the other runtimes already have.

How to reproduce it

  1. Create a CacheRuntime whose worker declares a non-zero tiered store:
spec:
  worker:
    replicas: 2
    tieredStore:
      levels:
      - emptyDir: {quota: 1Gi}
        high: "0.8"
        low: "0.5"
  1. Wait for the worker pods to be scheduled, then read the labels off a node running one:
kubectl get node <node> -o json | jq '.metadata.labels | with_entries(select(.key | contains("fluid.io")))'

Observed on both worker nodes:

fluid.io/dataset-num                        = 1
fluid.io/s-cache-default-mooncake-demo      = true
fluid.io/s-default-mooncake-demo            = true
fluid.io/s-h-cache-t-default-mooncake-demo  = 0B     <-- 1Gi was configured

There is no fluid.io/s-h-cache-m-... and no fluid.io/s-h-cache-d-... label at all — the loop that would
add them never runs.

Worth contrasting: the same quota is read correctly everywhere else. The bound Dataset reports the expected
total for 2 replicas of 1Gi:

NAME            UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE
mooncake-demo   2.00GiB          0B       2.00GiB          0.0                 ...

So the tiered store spec itself is fine; only the path that feeds the node labels sees an empty value.

Additional Information

A fix needs a conversion from RuntimeTieredStore to the legacy TieredStore that base.WithTieredStore consumes. Two things are worth agreeing on before implementing, because they are product decisions rather than mechanical ones:

  1. Medium mapping. RuntimeTieredStoreLevel (api/v1alpha1/cacheruntime_types.go:275) has no MediumType field. It describes the medium structurally instead, through ProcessMemory, EmptyDir and HostPath, whereas the legacy Level (api/v1alpha1/common.go:39) carries an explicit MEM/SSD/HDD enum. ProcessMemory and an EmptyDir with medium: Memory clearly map to MEM. HostPath has no way to say whether it is backed by SSD or HDD, and GetLevelStorageMap splits precisely on MEM versus SSD|HDD, so this needs a decision — default HostPath to one of them, or extend the API.

  2. Whether the client tier counts. tieredStore appears twice in the CacheRuntime spec: on the worker (cacheruntime_types.go:114) and on the client (cacheruntime_types.go:123). These labels feed worker placement during scheduling, so it should be settled explicitly whether client-side cache contributes to a node's advertised capacity.

Note this is a separate problem from the tiered store memory quota being dropped from the worker memory limit (#6166, fixed by #6167): that one concerns the container's memory accounting, this one concerns the node labels. They share the tieredStore spec field but neither fix depends on the other.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions