It seems like fllama loads model in every request

[woobin3069]

I was experimenting with my code using;

      final request = OpenAiRequest(
        maxTokens: 512,
        messages: chatMessages,
        numGpuLayers: 99,
        modelPath: kIsWeb ? "web_model_id" : modelPath,
        frequencyPenalty: 0.1,
        presencePenalty: 1.1,
        topP: 0.95,
        contextSize: 4096,
        temperature: 0.5,
        logger: (log) {
          // AppLogger.instance.d('[llama.cpp] $log');
        },
      );

      final responseController = StreamController<String>();
      int maxBufferSize = 4096;
      String previousResponse = "";

            void handleResponse(String response, bool done) {
        if (response.length > previousResponse.length) {
          String newContent = response.substring(previousResponse.length);
          if (newContent.length > maxBufferSize) {
            responseController.add(newContent.substring(0, maxBufferSize));
          } else {
            responseController.add(newContent);
          }
          previousResponse = response;
        }
        if (done) responseController.close();
      }

      if (kIsWeb) {
        _runningRequestId = await fllamaChatMlcWeb(
          request,
          (downloadProgress, loadProgress) {
            AppLogger.instance.d(
                'Download progress: $downloadProgress, Load progress: $loadProgress');
          },
          handleResponse,
        );
      } else {
        _runningRequestId = await fllamaChat(request, handleResponse);
      }

But it seems to load the model with every query.

[breitburg]

Yeah, same here, seems like it. I would love to have more manual control over when the model is loaded in RAM and when to offload it.

[gray-seven]

Please cache the model and release it when loading another one. Or, support a function to load/remove models manually.