feat: pre-messages / next version of download on demand #7371

Simon-Laux · 2025-10-29T20:53:40Z

feat: Remove "Download maximum available until" and remove stock string DC_STR_DOWNLOAD_AVAILABILITY #7369
remove: partial downloads (remove creation of the stub messages) #7373
feat: pre messages can now get read receipts #7433
feat: send pre-message on messages with large attachments #7410
feat: receive pre-messages and adapt download on demand #7431
... for now read the issue for more info on todo items and what this is about

This is the branch for #7367
closes #7367

Currently removed tests

test_something.py::test_reaction_to_partially_fetched_msg
python/tests/test_1_online.py::test_webxdc_download_on_demand
src/calls/calls_tests.py::test_no_partial_calls
src/message/message_tests.rs::test_markseen_not_downloaded_msg
src/receive_imf/receive_imf_tests.rs::test_download_later
src/receive_imf/receive_imf_tests.rs::test_create_group_with_big_msg
src/receive_imf/receive_imf_tests.rs::test_prefer_references_to_downloaded_msgs
src/receive_imf/receive_imf_tests.rs::test_partial_download_key_contact_lookup
src/webxdc/webxdc_tests.rs::test_webxdc_update_for_not_downloaded_instance
src/download.rs::test_download_limit
src/download.rs::test_partial_receive_imf
src/download.rs::test_partial_download_and_ephemeral
src/download.rs::test_status_update_expands_to_nothing
src/download.rs::test_mdn_expands_to_nothing
src/download.rs::test_partial_download_trashed
src/reaction.rs::test_partial_download_and_reaction

Progress of the tests

Overview about the recycled(♻️) and dropped(🗑️) tests of the tests that I removed in #7373

🗑️test_reaction_to_partially_fetched_msg
🗑️ test_partial_download_and_ephemeral
🗑️ test_status_update_expands_to_nothing
♻️ test_partial_download_trashed
♻️ test_partial_download_and_reaction
♻️ test_partial_group_consistency
🗑️ test_partial_download_key_contact_lookup
♻️ test_webxdc_update_for_not_downloaded_instance
🗑️test_no_partial_calls
♻️ test_markseen_not_downloaded_msg
♻️ test_msg_seen_on_imap_when_downloaded
♻️ test_create_group_with_big_msg
♻️ test_download_later
🗑️ test_prefer_references_to_downloaded_msgs
♻️test_download_on_demand
♻️test_download_limit_chat_assignment
🗑️test_webxdc_download_on_demand

All the recycled tests were already recycled/re-made except for some which are still to do:
(none)

Furthermore there need to be new tests to test the downloading/scheduling changes.

TODO Tests

postponed to do tests

(none)

discarded/ignored test ideas

maybe we should reconsider / discuss those?

test that full message, replacing pre-message does not work if the message encryption state is not the same for both messages (both full- and pre-message)
- unencrypted pre- and full-message are not sent

src/sql/migrations.rs

Hocuri · 2025-12-18T15:24:35Z

src/sql/migrations.rs

+            "CREATE TABLE download_new (
+                rfc724_mid TEXT NOT NULL DEFAULT '',
+                msg_id INTEGER NOT NULL DEFAULT 0
+            ) STRICT;


@link2xt I'm wondering whether we should add UNIQUE(rfc724_mid) or UNIQUE(msg_id)? Or even both? I guess it doesn't really make sense to have the same message in there twice?

Do we need both msg_id and rfc724_mid in this table? rfc724_mid can be obtained from msg_id.
EDIT: Got it, msg_id is 0 when rows are inserted to download in fetch_new_msg_batch(). But UNIQUE(rfc724_mid) may still make sense.

EDIT: Added Make {download,available_post_msgs}.rfc724_mid columns PRIMARY KEY

iequidoo · 2025-12-22T06:47:30Z

deltachat-ffi/deltachat.h

 * - `download_limit` = Messages up to this number of bytes are downloaded automatically.
- *                    For larger messages, only the header is downloaded and a placeholder is shown.
+ *                    For messages with large attachments, two messages are sent:
+ *                    a Pre-Message containing metadata and a Post-Message containing the attachment.


"Full message" was a better term, "post-message" is literally "smth going after the message". And the full message should contain metadata as well, but the documentation doesn't say that

We know "post-" message is technically wrong, but it is unambiguous - nobody else is going to call anything else a "post-message".

I think "full message" wasn't a good suggestion because it can also be applied to "normal" small messages. I'd say "big message" is a good term, and Chat-Is-Big-Message also sounds good.

EDIT: Still need to improve the documentation here... Done.

deltachat-ffi/deltachat.h

deltachat-jsonrpc/src/api/types/message.rs

src/chat.rs

src/download.rs

src/chat.rs

src/download/pre_msg_metadata.rs

src/scheduler.rs

src/sql/migrations.rs

src/stock_str.rs

src/tests.rs

iequidoo · 2025-12-27T07:23:51Z

src/imap.rs

+                    uids_fetch.push(uid);
+                    uid_message_ids.insert(uid, message_id);


if download_limit.is_none_or(|download_limit| size <= download_limit) { uids_fetch.push(uid); uid_message_ids.insert(uid, message_id); } else { download_later.push(message_id.clone()); largest_uid_skipped = Some(uid); }

Maybe do smth like this here? This way pre-messages should still be downloaded immediately and big emails are downloaded later to speed up downloading of pre-messages and other small messages.

I've chacked that JSON-RPC tests work with this suggestion.

src/download.rs

src/message.rs

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <[email protected]> Co-authored-by: iequidoo <[email protected]>

…essage before There is a bug in pre-messages: - A pre-message adds an entry to the `msgs` table with the `rfc724_mid` of the post-message - If the pre-message and post-message are fetched in separate cycles: - `prefetch_should_download()` returns false, because `rfc724_mid_exists()` returns something - so, the message is not added to `download_later`, and never automatically downloaded.

…e-messages modify group members There are no partial messages anymore, so i have no idea what else to check.

Also rename pre_message_is_downloaded_for() to msg_is_downloaded_for(). It's also used to check if a post-message is downloaded.

It currently fails because pre-messages aren't deleted on IMAP.

src/mimefactory.rs

Hyphen shouldn't be surrounded by spaces in general.

…th pre-message ones When inserting pre-message checks in the middle, if they evaluate to false, the remaining checks aren't performed. While this may be correct currently, it's hard to maintain. Better move the pre-message checks into a separate block.

…_pre_message() Now it goes before `scan_folders()`. Scanning folders isn't that important, it's even debounced to once per minute. Before, `download_msgs()` even preceded the whole `fetch_idle()`.

…s as seen on IMAP

It's the post-message metadata by its nature and it's even put into the Chat-Post-Message-Metadata header.

iequidoo · 2026-01-04T18:42:47Z

https://github.com/chatmail/core/actions/runs/20696010931/job/59411461313?pr=7371
14 CFFI Python tests failed this way:

    def test_status(acfactory):
        """Test that status is transferred over the network."""
>       ac1, ac2 = acfactory.get_online_accounts(2)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/test_1_online.py:1200: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py/lib/python3.14/site-packages/deltachat/testplugin.py:543: in get_online_accounts
    accounts = [self.new_online_configuring_account(cache=True) for i in range(num)]
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py/lib/python3.14/site-packages/deltachat/testplugin.py:515: in new_online_configuring_account
    self._acsetup.add_configured(ac)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <deltachat.testplugin.ACSetup object at 0x7f5398a253b0>
account = <Account path=/tmp/pytest-of-runner/pytest-0/popen-gw3/test_status0/ac2/dc.db>

    def add_configured(self, account):
        """add an already configured account."""
>       assert account.is_configured()
E       assert False
E        +  where False = is_configured()
E        +    where is_configured = <Account path=/tmp/pytest-of-runner/pytest-0/popen-gw3/test_status0/ac2/dc.db>.is_configured

.tox/py/lib/python3.14/site-packages/deltachat/testplugin.py:294: AssertionError
----------------------------- Captured stdout call -----------------------------
CACHE HIT for ci-sdkjgf@ci-chatmail.testrun.org
[acsetup] 0.014 added already configured account <Account path=/tmp/pytest-of-runner/pytest-0/popen-gw3/test_status0/ac1/dc.db> ci-sdkjgf@ci-chatmail.testrun.org
CACHE HIT for ci-zdcpw2@ci-chatmail.testrun.org

It's the first time i see this, probably it just makes sense to port all tests to JSON-RPC.

…re-messages

iequidoo · 2026-01-04T21:04:52Z

Btw, TZ=0 DCC_MIME_DEBUG=1 RUST_BACKTRACE=1 RUST_LOG=trace cargo nextest run --locked --no-fail-fast build time increased for me by ~15.4% here as compared to main. Tests run time almost hasn't changed however. Would be good to find the source of this pessimization. rustc 1.92.0.
...
Have made more experiments. Min. build time, pre-messages vs main: +3.1%. Pre-messages vs pre-messages with commented out tests using include_bytes! (6 tests total): +1.65%. So maybe the pessimization was subjective. But maybe we should avoid include_bytes!, +1.65% looks significant.

Simon-Laux changed the title ~~[Branch] Pre-messages / next version of download on demand~~ [Branch] pre-messages / next version of download on demand Oct 29, 2025

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

link2xt force-pushed the pre-messages branch 3 times, most recently from 2f1c383 to 6bcc795 Compare December 11, 2025 12:00

link2xt mentioned this pull request Dec 11, 2025

feat: receive pre-messages and adapt download on demand #7431

Merged

26 tasks

link2xt force-pushed the pre-messages branch 6 times, most recently from 8443b17 to 702771f Compare December 11, 2025 21:24

Hocuri reviewed Dec 11, 2025

View reviewed changes

src/sql/migrations.rs Outdated Show resolved Hide resolved

link2xt force-pushed the pre-messages branch 2 times, most recently from bb0fd7c to a98fe05 Compare December 14, 2025 17:54

link2xt changed the title ~~[Branch] pre-messages / next version of download on demand~~ feat: pre-messages / next version of download on demand Dec 14, 2025

link2xt force-pushed the main branch from e0ae864 to 1ea8ed6 Compare December 14, 2025 19:44

Hocuri reviewed Dec 18, 2025

View reviewed changes

iequidoo reviewed Dec 22, 2025

View reviewed changes

iequidoo reviewed Dec 24, 2025

View reviewed changes

iequidoo reviewed Dec 27, 2025

View reviewed changes

iequidoo reviewed Dec 28, 2025

View reviewed changes

src/download.rs Outdated Show resolved Hide resolved

src/download.rs Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

iequidoo force-pushed the pre-messages branch 2 times, most recently from 4a119fc to 84ce946 Compare December 30, 2025 04:30

This comment was marked as resolved.

Sign in to view

iequidoo reviewed Dec 30, 2025

View reviewed changes

src/message.rs Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

iequidoo force-pushed the pre-messages branch from 4498e6c to 18346a1 Compare January 2, 2026 19:56

Hocuri and others added 6 commits January 2, 2026 17:01

Add a replacement for test_partial_group_consistency checking that pr…

db52f32

…e-messages modify group members There are no partial messages anymore, so i have no idea what else to check.

review comments

69209d5

Also rename pre_message_is_downloaded_for() to msg_is_downloaded_for(). It's also used to check if a post-message is downloaded.

test_msg_text_on_lost_pre_msg

a45bb4c

Add test_pre_and_post_msgs_deleted

8c614ab

It currently fails because pre-messages aren't deleted on IMAP.

Add test_post_msg_bad_sender

77ab447

iequidoo force-pushed the pre-messages branch from 18346a1 to 77ab447 Compare January 2, 2026 20:01

iequidoo added 2 commits January 2, 2026 17:41

run ruff check --fix

fdd7dd3

ignore test_pre_and_post_msgs_deleted

25e124a

iequidoo reviewed Jan 2, 2026

View reviewed changes

src/mimefactory.rs Outdated Show resolved Hide resolved

Add test_lost_pre_msg_vs_new_member

14a578e

iequidoo force-pushed the pre-messages branch from 09e754e to 5187646 Compare January 3, 2026 08:49

iequidoo added 2 commits January 3, 2026 15:52

fix: Delete pre-messages on IMAP

57f7d2c

Make {download,available_post_msgs}.rfc724_mid columns PRIMARY KEY

81d40fb

iequidoo force-pushed the pre-messages branch from 5187646 to 81d40fb Compare January 3, 2026 18:52

iequidoo added 3 commits January 3, 2026 17:00

Replace '-' with en dash for file size separation

f1b758a

Hyphen shouldn't be surrounded by spaces in general.

Make PreMessageMode usable without Option

76ee904

iequidoo force-pushed the pre-messages branch from 09d684c to 98b2e78 Compare January 3, 2026 20:27

link2xt and others added 2 commits January 3, 2026 20:43

Fix python lint

e728f40

Move download_msgs() right after download_known_post_messages_without…

0c3fc91

…_pre_message() Now it goes before `scan_folders()`. Scanning folders isn't that important, it's even debounced to once per minute. Before, `download_msgs()` even preceded the whole `fetch_idle()`.

This comment was marked as resolved.

Sign in to view

iequidoo added 2 commits January 4, 2026 13:39

fix: prefetch_should_download: Mark already existing outgoing message…

8ce649a

…s as seen on IMAP

Rename PreMsgMetadata to PostMsgMetadata

f45fe45

It's the post-message metadata by its nature and it's even put into the Chat-Post-Message-Metadata header.

Improve download_limit docs

c7164f1

iequidoo marked this pull request as ready for review January 4, 2026 19:44

This comment was marked as resolved.

Sign in to view

test_download_limit_chat_assignment: Work around lost and reordered p…

31a82cd

…re-messages

		uids_fetch.push(uid);
		uid_message_ids.insert(uid, message_id);

Uh oh!

feat: pre-messages / next version of download on demand #7371

Are you sure you want to change the base?

feat: pre-messages / next version of download on demand #7371

Conversation

Simon-Laux commented Oct 29, 2025 • edited by iequidoo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Currently removed tests

Progress of the tests

TODO Tests

postponed to do tests

discarded/ignored test ideas

Uh oh!

Uh oh!

Hocuri Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iequidoo Dec 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iequidoo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

link2xt Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

iequidoo Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iequidoo Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

iequidoo commented Jan 4, 2026

Uh oh!

This comment was marked as resolved.

iequidoo commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Simon-Laux commented Oct 29, 2025 •

edited by iequidoo

Loading

Hocuri Dec 18, 2025 •

edited

Loading

iequidoo Dec 20, 2025 •

edited

Loading

iequidoo Jan 4, 2026 •

edited

Loading

iequidoo Dec 27, 2025 •

edited

Loading

iequidoo commented Jan 4, 2026 •

edited

Loading