⚡️ Speed up function _extract_exception_from_message by 92% in PR #1120 (comparator-wrapped-exceptions)

[codeflash-ai]

⚡️ This pull request contains optimizations for PR #1120

If you approve this dependent PR, these changes will be merged into the original PR branch comparator-wrapped-exceptions.

This PR will be automatically closed if the original PR is merged.

---

📄 92% (0.92x) speedup for _extract_exception_from_message in codeflash/verification/comparator.py

⏱️ Runtime : 1.99 milliseconds → 1.04 milliseconds (best of 208 runs)

📝 Explanation and details

The optimized code achieves a 92% speedup through two key optimizations:

1. Precompiled Regex Pattern (Primary Optimization)

The original code compiled the regex pattern r"got (\w+)\(['\"]" on every function call using re.search(). The optimized version precompiles this pattern once at module load time as _RE_GOT_EXC = re.compile(r"got (\w+)\(['\"]").

Why this matters: Regex compilation involves parsing the pattern string, building an internal state machine, and allocating memory structures. Line profiler shows the original re.search() call taking 4.39ms (48.7% of total time), while the optimized precompiled search takes only 1.25ms (23.8%) - a 71% reduction in just this line.

2. Module-level Import

Moving import builtins from inside the function to module scope eliminates repeated import lookups. While Python caches imports in sys.modules, the original code still incurred overhead finding and loading the cached module on each call (1.36ms, 15.1% of runtime). The optimized version eliminates this entirely.

Performance Impact by Test Case Type

The annotated tests show the optimization is most effective for:

• High-frequency matching scenarios: Tests with valid exception patterns (e.g., ValueError, TypeError) see 74-90% speedup
• No-match scenarios: Empty strings or non-matching patterns see 127-150% speedup (microseconds → nanoseconds range)
• Bulk operations: The 1000-iteration stress test improves from 1.45ms → 712μs (104% faster)
• Large messages: Performance gains of 39-40% even with 10K+ character strings, showing the optimization scales well

Call Context Impact

Based on function_references, this function is called from _get_wrapped_exception() which handles exception unwrapping. Since exception handling can occur in hot paths (loops, recursive calls, error recovery flows), and the function may be called multiple times per exception chain, the optimization compounds significantly in real-world usage where hundreds or thousands of exceptions might be processed.

The optimization is particularly valuable when processing many exceptions (as shown in bulk test cases improving by 81-104%), which aligns with typical usage in verification/comparison workflows where exception matching happens repeatedly.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 1176 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 1 Passed
📊 Tests Coverage	100.0%

🌀 Click to see Generated Regression Tests

# imports
import pytest  # used for our unit tests

from codeflash.verification.comparator import _extract_exception_from_message


def test_basic_single_quote_match_returns_value_error_instance():
    # Basic scenario: message contains "got ValueError('...')", should return a ValueError instance
    msg = "Runtime: something bad happened, got ValueError('invalid input') while processing"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.56μs -> 3.12μs (78.5% faster)


def test_basic_double_quote_match_returns_key_error_instance():
    # Basic scenario with double quotes: ensure the regex handles both quote types
    msg = 'Handler failed - got KeyError("missing_key") - aborting'
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.16μs -> 2.91μs (77.6% faster)


def test_no_pattern_returns_none():
    # When there's no "got <ExceptionName>('..." pattern, function must return None
    msg = "This message mentions no wrapped exception at all."
    codeflash_output = _extract_exception_from_message(msg)  # 1.66μs -> 732ns (127% faster)


def test_non_builtin_exception_name_returns_none():
    # If the extracted name is not a builtin exception, the function must return None
    msg = "Wrapper: got CustomError('this is custom') in outer layer"
    codeflash_output = _extract_exception_from_message(msg)  # 6.25μs -> 4.62μs (35.4% faster)


def test_attribute_found_but_not_type_returns_none():
    # builtins has attributes that are not classes (e.g., True). Ensure they are rejected.
    msg = "Edge: got True('weird') observed"
    # getattr(builtins, 'True') exists but is not a type, so should return None
    codeflash_output = _extract_exception_from_message(msg)  # 5.12μs -> 2.65μs (92.9% faster)


def test_builtin_type_but_not_exception_returns_none():
    # Some builtin types exist but are not subclasses of BaseException, e.g., object
    msg = "got object('x') during parsing"
    codeflash_output = _extract_exception_from_message(msg)  # 4.90μs -> 2.69μs (82.5% faster)


def test_first_match_taken_when_multiple_occurrences():
    # When multiple "got ..." patterns exist, re.search should find the first one
    msg = "prefix got IndexError('i') middle got KeyError('k') suffix"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.20μs -> 2.93μs (77.7% faster)


def test_no_space_after_got_does_not_match():
    # The regex requires "got " (with space). Ensure "gotValueError" won't match.
    msg = "unexpected token gotValueError('x')"
    codeflash_output = _extract_exception_from_message(msg)  # 1.65μs -> 732ns (126% faster)


def test_long_message_with_late_exception_is_handled():
    # Large message where the pattern appears near the end; ensure performance and correct match.
    prefix = "x" * 10000  # long prefix but still reasonable in size
    msg = prefix + " got OSError('disk full')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 8.58μs -> 6.10μs (40.6% faster)


def test_multiple_calls_return_distinct_instances():
    # Each call should create a fresh exception instance (not reuse the same object)
    msg = "fatal: got SystemExit('stop')"
    codeflash_output = _extract_exception_from_message(msg)
    a = codeflash_output  # 5.41μs -> 2.94μs (83.7% faster)
    codeflash_output = _extract_exception_from_message(msg)
    b = codeflash_output  # 2.35μs -> 1.25μs (87.9% faster)


def test_empty_string_returns_none():
    # Edge case: empty input must safely return None rather than raising
    codeflash_output = _extract_exception_from_message("")  # 1.53μs -> 621ns (147% faster)


def test_non_str_input_raises_type_error():
    # The function expects a string; passing None should raise TypeError from re.search
    with pytest.raises(TypeError):
        _extract_exception_from_message(None)  # 3.89μs -> 2.84μs (37.1% faster)


def test_large_scale_many_messages_all_match():
    # Large-scale scenario: process many messages (under 1000) to check scalability.
    # We'll create 500 messages that should all match ValueError.
    count = 500
    msgs = [f"iteration {i}: got ValueError('v{i}')" for i in range(count)]
    # Validate all produced exceptions are ValueError instances
    results = [_extract_exception_from_message(m) for m in msgs]


def test_match_with_numeric_and_underscore_names_rejected_if_not_builtin():
    # Regex allows digits and underscores in the name; but non-builtin names should be rejected.
    msg = "got Custom_Error123('x')"
    codeflash_output = _extract_exception_from_message(msg)  # 6.52μs -> 4.65μs (40.3% faster)


def test_baseexception_name_returns_baseexception_instance():
    # If the matched name is BaseException itself, it's a subclass of BaseException and should be instantiated
    msg = "wrapper: got BaseException('root')"
    codeflash_output = _extract_exception_from_message(msg)
    res = codeflash_output  # 5.01μs -> 2.81μs (77.9% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

import pytest

from codeflash.verification.comparator import _extract_exception_from_message


def test_extract_value_error_with_single_quotes():
    """Test extraction of ValueError with single quotes in message."""
    msg = "got ValueError('invalid value')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.26μs -> 2.90μs (81.6% faster)


def test_extract_type_error_with_double_quotes():
    """Test extraction of TypeError with double quotes in message."""
    msg = 'got TypeError("type mismatch")'
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.16μs -> 2.79μs (84.6% faster)


def test_extract_runtime_error():
    """Test extraction of RuntimeError."""
    msg = "got RuntimeError('runtime failed')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.20μs -> 2.85μs (82.1% faster)


def test_extract_attribute_error():
    """Test extraction of AttributeError."""
    msg = "got AttributeError('no attribute')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.18μs -> 2.79μs (85.3% faster)


def test_extract_key_error():
    """Test extraction of KeyError."""
    msg = "got KeyError('key not found')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 4.98μs -> 2.83μs (76.3% faster)


def test_extract_index_error():
    """Test extraction of IndexError."""
    msg = "got IndexError('index out of range')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.25μs -> 2.85μs (84.5% faster)


def test_extract_zero_division_error():
    """Test extraction of ZeroDivisionError."""
    msg = "got ZeroDivisionError('division by zero')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.06μs -> 2.92μs (73.6% faster)


def test_extract_assertion_error():
    """Test extraction of AssertionError."""
    msg = "got AssertionError('assertion failed')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.04μs -> 2.88μs (74.7% faster)


def test_message_with_exception_pattern_in_middle():
    """Test extraction when pattern appears in middle of message."""
    msg = "some text before got ValueError('error') and more text after"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.07μs -> 2.92μs (73.9% faster)


def test_no_exception_pattern_returns_none():
    """Test that message without exception pattern returns None."""
    msg = "This is just a regular error message with no pattern"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 1.67μs -> 711ns (135% faster)


def test_empty_string_returns_none():
    """Test that empty string returns None."""
    msg = ""
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 1.55μs -> 622ns (150% faster)


def test_pattern_with_no_quote_returns_none():
    """Test that pattern without quotes returns None."""
    msg = "got ValueError()"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 2.65μs -> 1.74μs (51.7% faster)


def test_pattern_with_single_quote_no_closing():
    """Test pattern with single quote but no closing quote still matches opening quote."""
    msg = "got ValueError('"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.32μs -> 2.86μs (86.3% faster)


def test_pattern_with_double_quote_no_closing():
    """Test pattern with double quote but no closing quote still matches opening quote."""
    msg = 'got TypeError("'
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.14μs -> 2.77μs (85.2% faster)


def test_non_exception_type_returns_none():
    """Test that non-exception types in builtins return None."""
    msg = "got dict('something')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 4.91μs -> 2.58μs (89.9% faster)


def test_non_existent_type_returns_none():
    """Test that non-existent types return None."""
    msg = "got NonExistentError('error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 6.20μs -> 4.40μs (41.0% faster)


def test_case_sensitive_exception_name():
    """Test that exception names are case-sensitive."""
    msg = "got valueerror('lowercase error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.84μs -> 4.17μs (40.1% faster)


def test_case_sensitive_exception_name_uppercase():
    """Test that fully uppercase exception names are not valid."""
    msg = "got VALUEERROR('uppercase error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.83μs -> 4.15μs (40.6% faster)


def test_exception_name_with_underscore():
    """Test extraction of exceptions with underscores like OSError."""
    msg = "got OSError('os error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.28μs -> 2.94μs (79.9% faster)


def test_exception_name_with_multiple_underscores():
    """Test extraction of exceptions with multiple underscores."""
    msg = "got SystemExit('exit')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.31μs -> 2.94μs (80.3% faster)


def test_message_with_newlines():
    """Test message containing newline characters."""
    msg = "Error occurred:\ngot ValueError('invalid')\nmore details"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.09μs -> 2.89μs (76.4% faster)


def test_message_with_tabs():
    """Test message containing tab characters."""
    msg = "Error\t\tgot RuntimeError('runtime issue')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.04μs -> 2.79μs (80.3% faster)


def test_message_with_special_characters_in_error_message():
    """Test message with special regex characters in the error message part."""
    msg = "got ValueError('error.*with+special$chars')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.20μs -> 2.87μs (81.5% faster)


def test_message_with_escaped_quotes():
    """Test message with escaped quotes in the error text."""
    msg = 'got ValueError("error with \\"escaped\\" quotes")'
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.19μs -> 2.79μs (85.7% faster)


def test_multiple_exception_patterns_extracts_first():
    """Test that when multiple patterns exist, the first one is extracted."""
    msg = "got ValueError('first') and got TypeError('second')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.07μs -> 2.79μs (82.0% faster)


def test_exception_name_with_numbers():
    """Test exception name containing numbers."""
    msg = "got Error1('test')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 6.20μs -> 4.31μs (43.9% faster)


def test_very_long_exception_name():
    """Test very long exception name that doesn't exist."""
    msg = "got " + "A" * 100 + "('error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 6.63μs -> 4.78μs (38.8% faster)


def test_zero_length_exception_name():
    """Test pattern with empty exception name (only space between 'got' and '(')."""
    # This won't match the regex pattern at all
    msg = "got ('error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 2.20μs -> 1.26μs (74.5% faster)


def test_exception_name_with_leading_space():
    """Test that leading space in exception name doesn't match."""
    msg = "got  ValueError('error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 2.15μs -> 1.28μs (68.0% faster)


def test_import_error_extraction():
    """Test extraction of ImportError."""
    msg = "got ImportError('module not found')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.46μs -> 3.07μs (78.1% faster)


def test_not_implemented_error_extraction():
    """Test extraction of NotImplementedError."""
    msg = "got NotImplementedError('not yet implemented')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.24μs -> 2.90μs (80.4% faster)


def test_unicode_error_extraction():
    """Test extraction of UnicodeError."""
    msg = "got UnicodeError('unicode issue')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.19μs -> 2.81μs (84.9% faster)


def test_value_error_is_base_exception():
    """Test that extracted exception is instance of BaseException."""
    msg = "got ValueError('test')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.16μs -> 2.88μs (79.4% faster)


def test_message_with_multiple_word_context():
    """Test extraction in context with multiple words."""
    msg = "The computation failed because got ValueError('computation error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.08μs -> 2.67μs (89.9% faster)


def test_exception_at_start_of_message():
    """Test exception pattern at the very start of message."""
    msg = "got RuntimeError('start of message')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.05μs -> 2.85μs (77.5% faster)


def test_exception_at_end_of_message():
    """Test exception pattern at the very end of message."""
    msg = "Some error occurred, got NameError('"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.32μs -> 2.83μs (88.4% faster)


def test_large_message_with_exception_pattern():
    """Test extraction from very large message containing exception pattern."""
    # Create a large message with pattern in the middle
    large_text = "x" * 10000
    msg = large_text + " got ValueError('error in large message') " + large_text
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 8.29μs -> 5.94μs (39.4% faster)


def test_large_message_without_pattern():
    """Test that large message without pattern returns None efficiently."""
    msg = "x" * 50000 + " regular error message " + "y" * 50000
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 32.6μs -> 31.6μs (3.13% faster)


def test_many_similar_patterns_in_message():
    """Test message with many similar but invalid patterns."""
    # Create message with many partial patterns that don't match the regex
    msg = "error " * 500 + "got ValueError('actual error') " + "more " * 500
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 6.25μs -> 3.90μs (60.4% faster)


def test_exception_pattern_after_many_false_starts():
    """Test finding valid pattern after many instances of 'got' without exception."""
    msg = "got this and that, got here, got there, " * 100 + "got TypeError('real error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 25.2μs -> 22.6μs (11.8% faster)


def test_many_builtin_exception_types():
    """Test extraction of various builtin exception types sequentially."""
    exception_types = [
        ValueError,
        TypeError,
        RuntimeError,
        AttributeError,
        KeyError,
        IndexError,
        ZeroDivisionError,
        AssertionError,
        ImportError,
        OSError,
        NameError,
        NotImplementedError,
    ]

    for exc_type in exception_types:
        msg = f"got {exc_type.__name__}('test')"
        codeflash_output = _extract_exception_from_message(msg)
        result = codeflash_output  # 25.4μs -> 14.0μs (81.8% faster)


def test_stress_with_repeated_calls():
    """Test function performance with repeated calls on same message."""
    msg = "got RuntimeError('stress test')"
    # Call function multiple times to test repeated processing
    for _ in range(1000):
        codeflash_output = _extract_exception_from_message(msg)
        result = codeflash_output  # 1.45ms -> 712μs (104% faster)


def test_stress_with_varied_messages():
    """Test function performance with many different messages."""
    exception_names = [
        "ValueError",
        "TypeError",
        "RuntimeError",
        "AttributeError",
        "KeyError",
        "IndexError",
        "ZeroDivisionError",
        "AssertionError",
        "ImportError",
        "OSError",
        "NameError",
        "NotImplementedError",
    ]

    # Create 100 varied messages with different exceptions
    messages = []
    for i in range(100):
        exc_name = exception_names[i % len(exception_names)]
        msg = f"error {i}: got {exc_name}('message {i}')"
        messages.append(msg)

    # Process all messages
    results = []
    for msg in messages:
        codeflash_output = _extract_exception_from_message(msg)
        result = codeflash_output  # 162μs -> 85.5μs (90.2% faster)
        results.append(result)


def test_message_with_long_exception_message_content():
    """Test extraction when the quoted error message is very long."""
    long_error_msg = "x" * 5000
    msg = f"got ValueError('{long_error_msg}')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.26μs -> 2.81μs (86.9% faster)


def test_message_with_alternating_quote_types():
    """Test messages with mixed quote types."""
    messages = [
        "got ValueError('error 1')",
        'got TypeError("error 2")',
        "got RuntimeError('error 3')",
        'got OSError("error 4")',
    ]

    for msg in messages:
        codeflash_output = _extract_exception_from_message(msg)
        result = codeflash_output  # 11.6μs -> 6.51μs (78.4% faster)


def test_extracted_exception_can_be_raised():
    """Test that extracted exception can actually be raised."""
    msg = "got ValueError('test value error')"
    codeflash_output = _extract_exception_from_message(msg)
    result = codeflash_output  # 5.17μs -> 2.91μs (77.9% faster)

    # Verify the exception can be raised
    with pytest.raises(ValueError):
        raise result


def test_extracted_exception_multiple_types():
    """Test that different messages extract different exception types."""
    msg1 = "got ValueError('error')"
    msg2 = "got TypeError('error')"

    codeflash_output = _extract_exception_from_message(msg1)
    result1 = codeflash_output  # 5.16μs -> 2.96μs (74.0% faster)
    codeflash_output = _extract_exception_from_message(msg2)
    result2 = codeflash_output  # 2.40μs -> 1.40μs (71.3% faster)


# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.

from codeflash.verification.comparator import _extract_exception_from_message


def test__extract_exception_from_message():
    _extract_exception_from_message("")

🔎 Click to see Concolic Coverage Tests

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
codeflash_concolic_hf5mz_7s/tmpfrmfcdtv/test_concolic_coverage.py::test__extract_exception_from_message	1.69μs	672ns	152%✅

To edit these changes git checkout codeflash/optimize-pr1120-2026-01-20T06.21.39 and push.