Email demo mbox by bmerkle · Pull Request #198 · microsoft/typeagent-py

bmerkle · 2026-02-15T22:53:52Z

added mbox handling

major refactoring
added mbox handling
added filters for first n, or last n mails of mbox
added filters for selecting mails with a date range, after or before
use verbose option to debug ingestion in detail
hardened mbox reading to cope with also invalid entries and other codecs, e.g.

testdata which i am using:

currently ingested fwts and bazzaar mbox.
unbunt mbox contain several 10k of mails hence we shall use selectors there (e.g. date ranges)

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          13/02/2026    21:18        2140627 bazaar-announce.mbox
-a---          13/02/2026    21:55         648468 fwts-announce.mbox
-a---          13/02/2026    20:19      269515931 kubuntu-users.mbox
-a---          13/02/2026    20:18      195869391 ubuntu-devel.mbox

- hardened mbox reading to cope with also invalid entries and other codecs, e.g. - ubuntu-devel.mbox and kubuntu-users.mbox contained kind of special mails, which resulted in errors without this hardening first.

- added mbox handling - added filters for first n, or last n mails of mbox - added filters for selecting mails with a date range, after or before - use verbose option to debug ingestion in detail

Copilot

Pull request overview

This PR adds mbox file support to the email ingestion tool, enabling processing of mbox mailbox files in addition to individual .eml files. The changes include filtering capabilities (first/last N emails, date ranges) and hardened encoding/header handling for robustness with real-world email data.

Changes:

Added mbox file format support with index-based and date-based filtering
Refactored argument parsing to use mutually exclusive groups for --eml and --mbox sources
Enhanced email header handling to properly coerce Header objects to strings

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 7 comments.

File	Description
tools/ingest_email.py	Added mbox support, filtering options (--first/--last/--after/--before), refactored email iteration logic into _iter_emails function
src/typeagent/emails/email_import.py	Added import_emails_from_mbox and count_emails_in_mbox functions, improved header type handling with _header_to_str, enhanced encoding error handling
.gitignore	Added /tests/testdata/email-mbox to ignore test data files

tools/ingest_email.py

src/typeagent/emails/email_import.py

tools/ingest_email.py

src/typeagent/emails/email_import.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

more memory efficient for large mbox Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

- added testcases

src/typeagent/emails/email_import.py

Two changes: 1. Default charset: or "utf-8" → or "latin-1" — when no charset header is present, latin-1 preserves all bytes. 2. LookupError fallback: same switch to "latin-1" (no errors param needed since latin-1 never fails on any byte value).

gvanrossum

I haven't looked at the tests yet, but let's first get these comments out of the way. :-)

tools/ingest_email.py

.gitignore

tests/test_mbox.py

tools/ingest_email.py

src/typeagent/emails/email_import.py

tests/testdata/email-mbox/bazaar-announce.mbox

tools/ingest_email.py

src/typeagent/emails/email_import.py

- mbox are expanded to eml - removed test data - fixed argument handling dates - removed first/last argument handling and replaced with limit etc

bmerkle · 2026-02-17T11:59:03Z

we could also remove the --eml switch again, because it is now the only way to import

gvanrossum

Somehow I can't seem to view test_mbox.py on this flight. I'll look into it later.

docs/demos.md

.gitignore

src/typeagent/emails/email_import.py

tools/gmail/mbox_dump.py

docs/demos.md

gvanrossum · 2026-02-17T15:49:48Z

we could also remove the --eml switch again, because it is now the only way to import

Yes please.

gvanrossum

Here's the review for test_mbox.py.

tests/test_mbox.py

- Updated .gitignore to reflect new email test data structure. - Modified demos.md to clarify usage of email ingestion tools and updated command-line arguments. - Changed references from `tools/gmail/` to `tools/mail/` in documentation and scripts. - Refactored test cases in test_mbox.py to use new date filtering parameters: --start-date and --stop-date. - Updated ingest_email.py to replace --after and --before with --start-date and --stop-date for date filtering. - Added new tools for downloading Gmail and Outlook emails as .eml files: gmail_dump.py and outlook_dump.py. - Introduced mbox_dump.py for extracting emails from mbox files into individual .eml files.

…in date filtering tests

bmerkle · 2026-02-17T19:50:05Z

all review points should be implemented now

gvanrossum

Bunch more nits. Getting pretty close!

I don't have the courage to look at outlook_dump.py yet. :-)

src/typeagent/emails/email_import.py

tools/mail/mbox_dump.py

tools/ingest_email.py

- improve date parsing logic - add option for output-dir similar to gmail client - remove outlook-dump.py as this is handeled in microsoft#199 (copilot added the file arbitraritly, grrr.)

bmerkle · 2026-02-19T10:41:52Z

outlook_dump.py

Copilot added the file arbitrarily and i did not notice it during commit.
Is removed now. we look at this in #199.

bmerkle added 4 commits February 15, 2026 23:33

- added mbox handling

5d0a61d

- hardened mbox reading to cope with also invalid entries and other codecs, e.g. - ubuntu-devel.mbox and kubuntu-users.mbox contained kind of special mails, which resulted in errors without this hardening first.

- major refactoring

47d34cb

- added mbox handling - added filters for first n, or last n mails of mbox - added filters for selecting mails with a date range, after or before - use verbose option to debug ingestion in detail

fix and update docs

853d1b9

fixed format

89dc931

Copilot AI review requested due to automatic review settings February 15, 2026 22:53

Copilot started reviewing on behalf of bmerkle February 15, 2026 22:54 View session

Copilot AI reviewed Feb 15, 2026

View reviewed changes

bmerkle and others added 5 commits February 16, 2026 00:01

Update tools/ingest_email.py

38e6282

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/typeagent/emails/email_import.py

a3d2002

more memory efficient for large mbox Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

added test data for small mbox

85b14b1

- fixed copilot reviews

b2028a2

- added testcases

semantic of before is usually inclusive, so >= is ok

8947349

gvanrossum reviewed Feb 16, 2026

View reviewed changes

src/typeagent/emails/email_import.py Outdated Show resolved Hide resolved

gvanrossum reviewed Feb 16, 2026

View reviewed changes

bmerkle added 3 commits February 16, 2026 20:50

removed 2 mbox data files

2a0335e

- implemented review feedback from Guido

969da13

- mbox are expanded to eml - removed test data - fixed argument handling dates - removed first/last argument handling and replaced with limit etc

fix: update email preview length in verbose output

360f7d7

gvanrossum reviewed Feb 17, 2026

View reviewed changes

tests/test_mbox.py Outdated Show resolved Hide resolved

tests/test_mbox.py Outdated Show resolved Hide resolved

tests/test_mbox.py Outdated Show resolved Hide resolved

tests/test_mbox.py Outdated Show resolved Hide resolved

bmerkle added 4 commits February 17, 2026 20:29

moved email date filtering logic and update related tests

62f7cb9

fix: remove carriage returns from email test cases for consistency

56184d7

fix: adjust naive timestamp handling to ensure consistent UTC offset …

eefc8d7

…in date filtering tests

gvanrossum reviewed Feb 18, 2026

View reviewed changes

gvanrossum mentioned this pull request Feb 19, 2026

- implementation of outlook email dump tool #199

Open

- enhance email header handling

892480d

- improve date parsing logic - add option for output-dir similar to gmail client - remove outlook-dump.py as this is handeled in microsoft#199 (copilot added the file arbitraritly, grrr.)

Comments

Conversation

bmerkle commented Feb 15, 2026 • edited by gvanrossum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

added mbox handling

testdata which i am using:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmerkle commented Feb 17, 2026

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gvanrossum commented Feb 17, 2026

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmerkle commented Feb 17, 2026

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bmerkle commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bmerkle commented Feb 15, 2026 •

edited by gvanrossum

Loading