Feat/huggingface backend by pmady · Pull Request #1665 · dragonflyoss/client

pmady · 2026-02-07T23:05:01Z

Feature

What does this PR do?

This PR adds support for downloading files from Hugging Face Hub repositories using the `hf://` URL scheme, addressing issue dragonflyoss/dragonfly#4419.

Features

New `hf://` URL scheme: Download models, datasets, and spaces from Hugging Face Hub
URL format: `hf://[repo_type/]/[/][@]`
Authentication: Support via `--hf-token` flag or `HF_TOKEN` environment variable
Git LFS support: Handles large model files through Hugging Face HTTP API
Repository listing: Supports recursive downloads with `-r` flag

Usage Examples

# Download a single file
dfget hf://deepseek-ai/DeepSeek-OCR/model.safetensors -O /tmp/model.safetensors

# Download entire repository
dfget hf://deepseek-ai/DeepSeek-OCR -O /tmp/DeepSeek-OCR/ -r

# With authentication for private repos
dfget hf://owner/private-repo/model.bin -O /tmp/model.bin --hf-token=<token>

Changes

`dragonfly-client-backend/src/huggingface.rs` (new): Hugging Face backend implementation
`dragonfly-client-backend/src/lib.rs`: Register `hf` backend in BackendFactory
`dragonfly-client-backend/Cargo.toml`: Add serde dependencies
`dragonfly-client/src/bin/dfget/main.rs`: Add `--hf-token` CLI argument and examples

Related Issues

Closes dragonflyoss/dragonfly#4419

Checklist

Code follows project style guidelines
Tests added/updated (46 tests pass)
Documentation updated (CLI help)
Commits are signed off"

Implement a new backend for downloading files from Hugging Face Hub repositories using the hf:// URL scheme. Features: - Support for models, datasets, and spaces repositories - URL parsing with revision/branch support (e.g., hf://owner/repo@v1.0) - Authentication via HF_TOKEN environment variable - Git LFS file support for large model files - Repository listing for recursive downloads Signed-off-by: pmady <pmady@users.noreply.github.com>

Register the hf:// scheme backend in load_builtin_backends() and update tests to include the new backend in expected backends list. Signed-off-by: pmady <pmady@users.noreply.github.com>

Add serde and serde_json workspace dependencies required for parsing Hugging Face API responses. Signed-off-by: pmady <pmady@users.noreply.github.com>

Add --hf-token argument for Hugging Face authentication and include usage examples in the CLI help documentation. Examples added: - Download single file: dfget hf://owner/repo/path -O /tmp/file - Download repository: dfget hf://owner/repo -O /tmp/repo/ -r - With authentication: dfget hf://... --hf-token=<token> Signed-off-by: pmady <pmady@users.noreply.github.com>

codecov · 2026-02-07T23:08:54Z

Codecov Report

❌ Patch coverage is 45.70637% with 392 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.66%. Comparing base (b71b4b5) to head (be0ef12).

Files with missing lines	Patch %	Lines
dragonfly-client-backend/src/hugging_face.rs	39.96%	344 Missing ⚠️
dragonfly-client/src/bin/dfget/main.rs	64.63%	29 Missing ⚠️
dragonfly-client/src/resource/persistent_task.rs	0.00%	6 Missing ⚠️
dragonfly-client/src/resource/task.rs	0.00%	6 Missing ⚠️
dragonfly-client-backend/src/object_storage.rs	79.16%	5 Missing ⚠️
dragonfly-client-backend/src/hdfs.rs	0.00%	1 Missing ⚠️
dragonfly-client/src/proxy/mod.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1665      +/-   ##
==========================================
- Coverage   50.86%   50.66%   -0.21%     
==========================================
  Files          84       85       +1     
  Lines       20523    21196     +673     
==========================================
+ Hits        10439    10738     +299     
- Misses      10084    10458     +374

Files with missing lines	Coverage Δ
dragonfly-client-backend/src/http.rs	`97.68% <100.00%> (+0.05%)`	⬆️
dragonfly-client-backend/src/lib.rs	`96.10% <100.00%> (+0.06%)`	⬆️
dragonfly-client-core/src/error/mod.rs	`59.09% <ø> (ø)`
dragonfly-client/src/announcer/mod.rs	`0.00% <ø> (ø)`
dragonfly-client/src/grpc/dfdaemon_download.rs	`4.67% <ø> (ø)`
dragonfly-client/src/grpc/dfdaemon_upload.rs	`0.00% <ø> (ø)`
dragonfly-client/src/resource/piece.rs	`57.94% <ø> (ø)`
dragonfly-client-backend/src/hdfs.rs	`40.90% <0.00%> (ø)`
dragonfly-client/src/proxy/mod.rs	`0.00% <0.00%> (ø)`
dragonfly-client-backend/src/object_storage.rs	`88.62% <79.16%> (ø)`
... and 4 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Apply cargo fmt to fix formatting in huggingface.rs and lib.rs. Signed-off-by: pmady <pmady@users.noreply.github.com>

pmady · 2026-02-07T23:24:15Z

Hi maintainers, could you please add the enhancement label to this PR? The PR Label check requires one of: bug, enhancement, documentation, or dependencies. Thank you!

gaius-qi · 2026-02-09T12:12:01Z

@pmady Thanks for contributing this PR. Can you add document for https://d7y.io/docs/next/operations/integrations/hugging-face/ and https://d7y.io/docs/next/reference/commands/client/dfget/#download-with-different-protocols.

d7y.io Repo: https://github.com/dragonflyoss/d7y.io

dragonfly-client-backend/src/huggingface.rs

Copilot

Pull request overview

Adds a new hf:// backend so dfdaemon/dfget can download from Hugging Face Hub repositories (models/datasets/spaces), including repo listing for recursive downloads, and introduces a CLI flag intended for HF authentication.

Changes:

Add huggingface backend implementation and register scheme hf in BackendFactory.
Extend dfget CLI help/examples and add --hf-token argument.
Add serde/serde_json deps for HF API response parsing.

Reviewed changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`dragonfly-client/src/bin/dfget/main.rs`	Adds HF usage examples and `--hf-token` CLI option.
`dragonfly-client-backend/src/lib.rs`	Registers the new `hf` backend and updates backend factory tests.
`dragonfly-client-backend/src/huggingface.rs`	Implements HF backend: URL parsing, stat/list/get/exists, plus unit tests.
`dragonfly-client-backend/Cargo.toml`	Adds serde dependencies needed by the new backend.
`Cargo.lock`	Locks new transitive deps from serde additions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dragonfly-client-backend/src/huggingface.rs

dragonfly-client-backend/src/lib.rs

- Update copyright year to 2026 - Remove environment variable fallback for HF_TOKEN, keep only CLI option - Implement ParsedHfUrl with TryFrom<Url> and TryFrom<&str> traits - Make ParsedHfUrl and RepoType public structs - Update tests to use TryFrom pattern Signed-off-by: pmady <pmady@users.noreply.github.com>

The HF backend was instantiated with HuggingFace::new() at startup, making the --hf-token CLI flag ineffective since the token was stored on the struct but never received from dfget. Changes: - dfget: inject --hf-token as Authorization header into request_header so it flows through gRPC to dfdaemon and into the backend - HF backend: remove stored token field, read auth from request http_header instead via build_headers() method - Remove new_with_token() constructor since it is no longer needed Signed-off-by: pmady <pmady@users.noreply.github.com>

- Fix URL parsing: remove redundant early-return branch, always require owner/repo (two segments) after optional type prefix - Fix list_files to return hf:// URLs instead of https:// so downstream downloads continue using the HF backend (preserving auth and semantics) - Use versioned DEFAULT_USER_AGENT matching the HTTP backend pattern (concat!("dragonfly", "/", env!("CARGO_PKG_VERSION"))) and allow user-supplied User-Agent to override it - Fix dataset test to use proper owner/repo URL format - Add comprehensive test coverage: dataset, space, explicit model type, invalid scheme, missing repo, build_hf_url, build_headers behavior Signed-off-by: pmady <pmady@users.noreply.github.com>

pmady · 2026-02-09T17:34:48Z

@gaius-qi I've created a documentation PR at dragonflyoss/d7y.io#386 that adds:

Hugging Face integration page: New section documenting the native hf:// protocol with URL format, single file download, authentication (--hf-token), recursive repository download, dataset download, and revision-specific download examples.
dfget reference page: Added "Download with Hugging Face protocol" section with complete examples.

…ngface-backend

…gFace proto support - Add HuggingFace proto message support in dfget for passing hf_token through the download request instead of injecting Authorization headers manually in convert_args. - Update dragonfly-api dependency from 2.2.13 to 2.2.19 Signed-off-by: Gaius <gaius.qi@gmail.com>

gaius-qi · 2026-02-11T05:49:07Z

@pmady Thanks, I'll finish the review by this week.

Signed-off-by: Gaius <gaius.qi@gmail.com>

…ngface-backend Signed-off-by: Gaius <gaius.qi@gmail.com>

…port - Rename huggingface.rs to hugging_face.rs to follow Rust naming conventions - Rewrite HuggingFace backend with improved architecture: - Inject Arc<Config> for TLS and hickory DNS configuration - Replace http_header-based auth with dedicated HuggingFace proto field - Add insecure_skip_verify flag via --hf-insecure-skip-verify CLI arg - Use TryStreamExt over StreamExt for proper error propagation - Remove unused Default impl and redundant helper methods - Add HuggingFace field to StatRequest, GetRequest, ExistsRequest, and PutRequest structs for passing auth tokens through the request pipeline - Propagate hugging_face field through task, persistent_task, piece, dfdaemon_download, and dfdaemon_upload request handling - Bump dragonfly-api to 2.2.20 for HuggingFace proto message support - Refactor backend modules to use explicit crate imports instead of super:: prefix for cleaner module boundaries - Update doc comments across backend modules to use imperative mood Signed-off-by: Gaius <gaius.qi@gmail.com>

Signed-off-by: Gaius <gaius.qi@gmail.com>

gaius-qi · 2026-03-05T14:14:28Z

@pmady The downloaded file is named LICENSE@main, but the @main part should be removed.

@revision

The downloaded file was named LICENSE@main instead of LICENSE because make_output_by_entry used the raw URL path which includes the @revision suffix. Strip the @revision from both the base URL path and entry URL path when the scheme is hf to produce correct output filenames. Signed-off-by: pmady <pmady@users.noreply.github.com>

pmady · 2026-03-05T15:06:48Z

@gaius-qi Fixed in be0ef12 — make_output_by_entry now strips the @revision suffix from HF URL paths so the output filename is LICENSE instead of LICENSE@main. Added tests for both default and custom revisions.

pmady added 4 commits February 7, 2026 16:42

feat(backend): register Hugging Face backend in BackendFactory

0d07910

Register the hf:// scheme backend in load_builtin_backends() and update tests to include the new backend in expected backends list. Signed-off-by: pmady <pmady@users.noreply.github.com>

chore(backend): add serde dependencies for Hugging Face backend

d73ef0b

Add serde and serde_json workspace dependencies required for parsing Hugging Face API responses. Signed-off-by: pmady <pmady@users.noreply.github.com>

github-actions bot requested review from fcgxz2003, gaius-qi and hyy0322 February 7, 2026 23:05

github-actions bot assigned pmady Feb 7, 2026

style: fix formatting issues

c2ff0ed

Apply cargo fmt to fix formatting in huggingface.rs and lib.rs. Signed-off-by: pmady <pmady@users.noreply.github.com>

gaius-qi added the enhancement New feature or request label Feb 9, 2026

gaius-qi requested changes Feb 9, 2026

View reviewed changes

dragonfly-client-backend/src/huggingface.rs Outdated Show resolved Hide resolved

dragonfly-client-backend/src/huggingface.rs Outdated Show resolved Hide resolved

dragonfly-client-backend/src/huggingface.rs Outdated Show resolved Hide resolved

gaius-qi requested a review from Copilot February 9, 2026 13:09

Copilot started reviewing on behalf of gaius-qi February 9, 2026 13:10 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

pmady added 3 commits February 9, 2026 10:24

pmady mentioned this pull request Feb 9, 2026

docs: add hf:// protocol documentation for Hugging Face integration dragonflyoss/d7y.io#386

Open

pmady requested a review from gaius-qi February 9, 2026 18:13

gaius-qi added 2 commits February 10, 2026 10:51

Merge branch 'main' of github.com:dragonflyoss/client into feat/huggi…

b18588c

…ngface-backend

gaius-qi added 3 commits February 14, 2026 12:01

docs(huggingface): update comments for clarity

1ede78c

Signed-off-by: Gaius <gaius.qi@gmail.com>

Merge branch 'main' of github.com:dragonflyoss/client into feat/huggi…

0da7f3a

…ngface-backend Signed-off-by: Gaius <gaius.qi@gmail.com>

Merge branch 'main' of github.com:dragonflyoss/client into feat/huggi…

a8debc5

…ngface-backend Signed-off-by: Gaius <gaius.qi@gmail.com>

gaius-qi force-pushed the feat/huggingface-backend branch from 915c64f to a8debc5 Compare March 4, 2026 06:10

gaius-qi added 3 commits March 4, 2026 22:49

test(hugging_face): remove invalid scheme test

d468424

Signed-off-by: Gaius <gaius.qi@gmail.com>

fix(error): improve error message for max file limit

9dc2c33

Signed-off-by: Gaius <gaius.qi@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/huggingface backend#1665

Feat/huggingface backend#1665
pmady wants to merge 17 commits intodragonflyoss:mainfrom
pmady:feat/huggingface-backend

pmady commented Feb 7, 2026

Uh oh!

codecov bot commented Feb 7, 2026 •

edited

Loading

Uh oh!

pmady commented Feb 7, 2026

Uh oh!

gaius-qi commented Feb 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmady commented Feb 9, 2026

Uh oh!

gaius-qi commented Feb 11, 2026

Uh oh!

gaius-qi commented Mar 5, 2026 •

edited

Loading

Uh oh!

pmady commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pmady commented Feb 7, 2026

What does this PR do?

Features

Usage Examples

Changes

Related Issues

Checklist

Uh oh!

codecov bot commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pmady commented Feb 7, 2026

Uh oh!

gaius-qi commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmady commented Feb 9, 2026

Uh oh!

gaius-qi commented Feb 11, 2026

Uh oh!

gaius-qi commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmady commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Feb 7, 2026 •

edited

Loading

gaius-qi commented Feb 9, 2026 •

edited

Loading

gaius-qi commented Mar 5, 2026 •

edited

Loading