[FEAT] User input URL, then content loaded to knowledge table

[szeyu]

Is your feature request related to a problem? Please describe.

Yeah, so in my program has to scrap the content in url then save into a txt file. Then only can upload to knowledge table

Describe the solution you'd like

besides uploading file, can allow the user to upload url link for the jamai to extract the data into its knowledge table.

Describe alternatives you've considered

scrap the content in url then save into a txt file. Then only can upload to knowledge table

Additional context

the idea is gotten from
https://github.com/embedchain/embedchain
it got 9k stars so this idea is quite good

[aicodecraft1004]

Happy to take this on.

At a high level, I propose adding first-class URL ingestion that feeds into the existing file - knowledge - table pipeline (same chunking/embedding/indexing), so we avoid duplicating ingest logic.
My plan is to do a quick pass over the current ingest path and then implement a URL loader step upfront:

1. URL validation + SSRF protections (block localhost/private IP ranges) + timeouts/size limits.
2. Fetch content (start with text/html, optionally text/plain).
3. Extract readable text (title + main content) and normalize to markdown/text.
4. Persist metadata (source_url, title, fetched_at, content_hash) and dedupe on content_hash.
5. Hand off to the current ingest entrypoint for knowledge-table insertion.

I can open a PR with an MVP (HTML) + guardrails.
should v1 scope to HTML/text only, or include PDF URLs too?