Skip to content

feat: add ModelScope backend support with modelscope:// protocol#1673

Open
pmady wants to merge 3 commits intodragonflyoss:mainfrom
pmady:feat/modelscope-backend
Open

feat: add ModelScope backend support with modelscope:// protocol#1673
pmady wants to merge 3 commits intodragonflyoss:mainfrom
pmady:feat/modelscope-backend

Conversation

@pmady
Copy link
Contributor

@pmady pmady commented Feb 9, 2026

What this PR does / why we need it:

Adds a new backend for downloading models and datasets from ModelScope Hub using the modelscope:// URL scheme, similar to the existing Hugging Face backend.

URL Format

modelscope://[<repo_type>/]<owner>/<repo>[/<path>][@<revision>]

Examples

# Download a model repository
dfget modelscope://inclusionAI/Ling-1T -O /tmp/ling-1t/ --recursive

# Download a single file
dfget modelscope://inclusionAI/Ling-1T/config.json -O /tmp/config.json

# Download with authentication
dfget modelscope://inclusionAI/Ling-1T/config.json -O /tmp/config.json --ms-token=<token>

# Download a dataset
dfget modelscope://datasets/damo/squad-zh/train.json -O /tmp/train.json

# Download from a specific revision
dfget modelscope://inclusionAI/Ling-1T/config.json@v1.0 -O /tmp/config.json

Features

  • Support for models and datasets repository types
  • File listing via ModelScope API (/api/v1/models/{id}/repo/files) for recursive downloads
  • File download via ModelScope API (/api/v1/models/{id}/repo?FilePath=...)
  • Authentication via --ms-token CLI flag (injected as Authorization: Bearer header)
  • Range download support
  • Default revision is master (matching ModelScope convention)

Which issue(s) this PR fixes:

Closes dragonflyoss/dragonfly#4420

Does this PR introduce a user-facing change?

Yes. Users can now download files and repositories from ModelScope Hub using dfget modelscope://... with P2P acceleration.

Add a new backend for downloading models and datasets from ModelScope Hub
(modelscope.cn) using the modelscope:// URL scheme.

URL format: modelscope://[<repo_type>/]<owner>/<repo>[/<path>][@<revision>]

Features:
- Support for models and datasets repository types
- File listing via ModelScope API for recursive downloads
- Authentication via --ms-token CLI flag
- Range download support
- Default revision is 'master' (matching ModelScope convention)

Closes dragonflyoss/dragonfly#4420

Signed-off-by: Pradumna Saraf <pradumnasaraf@gmail.com>
Signed-off-by: pmady <pmady@users.noreply.github.com>
@pmady
Copy link
Contributor Author

pmady commented Feb 9, 2026

Hi maintainers, could you please add the enhancement label to this PR? The PR Label check requires one of: bug, enhancement, documentation, or dependencies. Thank you!

@codecov
Copy link

codecov bot commented Feb 9, 2026

Codecov Report

❌ Patch coverage is 44.52055% with 405 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.26%. Comparing base (3c7d0d4) to head (cc744bf).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
dragonfly-client-backend/src/model_scope.rs 43.11% 376 Missing ⚠️
dragonfly-client/src/bin/dfget/main.rs 40.00% 12 Missing ⚠️
dragonfly-client/src/resource/persistent_task.rs 0.00% 6 Missing ⚠️
dragonfly-client/src/resource/task.rs 0.00% 6 Missing ⚠️
dragonfly-client/src/grpc/dfdaemon_download.rs 0.00% 2 Missing ⚠️
dragonfly-client/src/grpc/dfdaemon_upload.rs 0.00% 2 Missing ⚠️
dragonfly-client-backend/src/hugging_face.rs 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1673      +/-   ##
==========================================
- Coverage   50.46%   50.26%   -0.20%     
==========================================
  Files          85       86       +1     
  Lines       21224    21950     +726     
==========================================
+ Hits        10711    11034     +323     
- Misses      10513    10916     +403     
Files with missing lines Coverage Δ
dragonfly-client-backend/src/http.rs 97.73% <100.00%> (+0.04%) ⬆️
dragonfly-client-backend/src/lib.rs 96.30% <100.00%> (+0.20%) ⬆️
dragonfly-client/src/resource/piece.rs 57.94% <ø> (ø)
dragonfly-client-backend/src/hugging_face.rs 35.80% <0.00%> (-0.06%) ⬇️
dragonfly-client/src/grpc/dfdaemon_download.rs 4.66% <0.00%> (-0.01%) ⬇️
dragonfly-client/src/grpc/dfdaemon_upload.rs 0.00% <0.00%> (ø)
dragonfly-client/src/resource/persistent_task.rs 0.00% <0.00%> (ø)
dragonfly-client/src/resource/task.rs 7.26% <0.00%> (-0.08%) ⬇️
dragonfly-client/src/bin/dfget/main.rs 49.59% <40.00%> (-0.15%) ⬇️
dragonfly-client-backend/src/model_scope.rs 43.11% <43.11%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gaius-qi gaius-qi added the enhancement New feature or request label Feb 10, 2026
@gaius-qi
Copy link
Member

@pmady Thanks, I'll finish the review by this week.

gaius-qi added 2 commits March 9, 2026 22:10
…scope-backend

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi gaius-qi force-pushed the feat/modelscope-backend branch from e2b0d83 to cc744bf Compare March 9, 2026 14:32
@gaius-qi gaius-qi enabled auto-merge (squash) March 9, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Supports directly pulling repositories from ModelScope

3 participants