MLServer fails to load models from OCI image mounts on KServe

[Snomaan6846]

MLServer XGBoost OCI Model Loading Issue (KServe)

Description

MLServer encounters issues loading XGBoost models when deployed on KServe using OCI model images (model-car sidecar). The issue is specific to XGBoost's C++ backend, which fails to resolve model file paths provided via model-car mounts.

Environment

• MLServer Version: 1.7.0+
• Platform: KServe on Kubernetes
• Storage: OCI Image via model-car sidecar
• Affected Runtime: XGBoost

Steps to Reproduce

1. Deploy an InferenceService using OCI image storage:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: xgboost-model
spec:
  predictor:
    model:
      modelFormat:
        name: xgboost
      storageUri: oci://quay.io/my-registry/xgboost-model:latest
      runtime: mlserver

2. Observe MLServer logs during model initialization

Expected Behavior

XGBoost models should load successfully from the path mounted by the model-car sidecar (for example, /mnt/models).

Actual Behavior

XGBoost fails to load the model due to path resolution errors originating from its C++ backend.

Error observed:

XGBoostError: filesystem error: cannot make canonical path: No such file or directory [/mnt/models/model.json]

Although the file exists and is readable via Python, the native XGBoost loader fails:

import os
path = "/mnt/models/model.json"

os.path.exists(path)        # True
os.path.realpath(path)      # /proc/<pid>/root/...

# Python file access works
with open(path, "r") as f:
    f.read()  # ✅ Success

# XGBoost C++ loader fails
import xgboost as xgb
xgb.Booster(model_file=path)  # ❌ Fails

Root Cause

• The XGBoost C++ library attempts to canonicalize model paths
• Model-car sidecar mounts resolve to /proc/<pid>/root/... or similar bind-mount paths
• These paths are valid for Python file I/O but cannot be resolved by XGBoost's native filesystem handling

Impact

• ❌ Prevents XGBoost models from loading when using OCI image storage with model-car
• ❌ Blocks XGBoost usage in this deployment configuration

Additional Context

• The issue is specific to XGBoost's C++ implementation
• Model files are present and readable from within the container
• Python-based access works as expected

Labels

bug kserve deployment xgboost runtime