Add ProTeGi Prompt Optimizer by DanielDango · Pull Request #57 · ardoco/lissa

DanielDango · 2026-01-22T11:53:25Z

Adds the ProTeGi Prompt Optimizer by Pryzant et al. of their paper "Automatic Prompt Optimization with "Gradient Descent" and Beam Search".

…timization with "Gradient Descent" and Beam Search" of Pryzant et al. 2023

- add new redis cache replacement strategy

…n-module' into feature/add-protegi-optimization

Copilot

Pull request overview

This pull request adds the ProTeGi Prompt Optimizer, implementing an automatic prompt optimization approach based on gradient descent by Pryzant et al. (2023). The implementation transcribes the original Python code into the LiSSA framework and modifies the core optimization interface to return arrays of prompts representing all intermediate optimization states rather than just the final result.

Changes:

Adds AutomaticPromptOptimizer with gradient-based prompt refinement using textual gradients and beam search
Modifies PromptOptimizer interface to return String[] instead of String to capture all intermediate optimization states
Implements cache improvements including OpenAI model instance caching and configurable cache conflict resolution strategies

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java	New gradient-based optimizer implementation with textual gradients, synonym generation, and beam search
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/GradientOptimizerConfig.java	Configuration record for gradient optimizer parameters
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/PromptOptimizer.java	Interface changed to return String[] instead of String
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeOptimizer.java	Updated to return array of prompts from all iterations
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeFeedbackOptimizer.java	Updated to return array of prompts from all iterations
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/MockOptimizer.java	Updated to return String[] instead of String
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/OptimizerFactory.java	Added gradient optimizer case to factory method
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/ChatLanguageModelProvider.java	Added singleton caching for OpenAI chat model instances
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/cache/CacheReplacementStrategy.java	New enum defining strategies for cache conflict resolution
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/cache/RedisCache.java	Integrated conflict resolution strategy
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/cache/CacheManager.java	Updated to use new conflict resolution strategy
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Statistics.java	Added method to generate statistics for specific iteration counts
src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Optimization.java	Updated to handle array of prompts and generate statistics for each iteration
src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java	Updated to handle array of prompts and evaluate all iterations
src/test/java/edu/kit/kastel/sdq/lissa/ratlr/e2e/Requirement2RequirementE2ETest.java	Updated test to handle array return and select final prompt
src/test/resources/simplelogger.properties	Changed log level from debug to info
example-configs/gradient-optimizer-config.json	Example configuration file for gradient optimizer
src/test/resources/warc/WARC_gradient_gpt_gpt-4o-mini-2024-07-18_0.json	Test configuration file
src/test/resources/expected/GradientOptimizerExpectation.txt	Expected output for gradient optimizer test
src/test/resources/warc/cache/optimizerTests/gradient/*.json	Cached test responses

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Statistics.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeFeedbackOptimizer.java

src/test/resources/simplelogger.properties

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/ChatLanguageModelProvider.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

…d-protegi-optimization # Conflicts: # src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/cache/CacheManager.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/cache/RedisCache.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeFeedbackOptimizer.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeOptimizer.java # src/test/resources/simplelogger.properties

…n-module' into feature/add-protegi-optimization # Conflicts: # src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Optimization.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeFeedbackOptimizer.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeOptimizer.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/OptimizerFactory.java # src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/PromptOptimizer.java

…h protegi" This reverts commit d5f758b.

…timizer" This reverts commit 83b5b7b

…d-protegi-optimization

…ptimization

dfuchss

Some initial review

example-configs/gradient-optimizer-config.json

src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/ChatLanguageModelProvider.java

src/main/java/edu/kit/kastel/sdq/lissa/cli/command/OptimizeCommand.java

...it/kastel/sdq/lissa/ratlr/promptoptimizer/evaluator/UpperConfidenceBoundBanditEvaluator.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

dfuchss · 2026-02-19T14:55:44Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/GradientOptimizerConfig.java

+import edu.kit.kastel.sdq.lissa.ratlr.promptoptimizer.samplestrategy.SampleStrategy;
+import edu.kit.kastel.sdq.lissa.ratlr.promptoptimizer.samplestrategy.SamplerFactory;
+
+public record GradientOptimizerConfig(


Docs. Also document what the different prompts are for

dfuchss · 2026-02-19T14:57:25Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/PromptOptimizer.java

+            @Nullable ModuleConfiguration configuration,
+            Set<TraceLink> goldStandard,
+            Metric metric,
+            Evaluator evaluator) {


Nullable evaluator I guess

The Evaluator is not (independently) nullable currently. Same as with the metric and goldstandard. The config null check may be extended to include any parameter, as the other parameters may be null iff config is also null.

- rename sampleAndEvaluate to selectAndEvaluate - add documentation - rework abstract base class as interface

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeFeedbackOptimizer.java

...u/kit/kastel/sdq/lissa/ratlr/promptoptimizer/promptselector/UpperConfidenceBoundBandits.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/GradientOptimizerConfig.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/promptselector/SimpleSelector.java

Copilot · 2026-03-05T11:20:19Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/LazyChatModel.java

+    private @Nullable ChatModel delegate;
+
+    /**
+     * Creates a new LazyChatModel with the specified supplier for the delegate.
+     *
+     * @param delegateSupplier The supplier that provides the chat model delegate
+     */
+    public LazyChatModel(Supplier<ChatModel> delegateSupplier) {
+        this.delegateSupplier = Objects.requireNonNull(delegateSupplier);
+    }
+
+    private ChatModel delegate() {
+        if (delegate != null) {
+            return delegate;
+        }
+        synchronized (this) {
+            if (delegate == null) {
+                delegate = delegateSupplier.get();
+            }
+            return delegate;
+        }


LazyChatModel uses double-checked locking, but delegate is not volatile. Without safe publication, another thread can observe a partially constructed delegate instance when reading delegate outside the synchronized block. Make delegate volatile (or synchronize all reads / use an AtomicReference/Supplier memoization) to ensure thread-safety matches the class Javadoc.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Optimization.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/AutomaticPromptOptimizer.java

docs/prompt-optimization.md

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/IterativeOptimizer.java

…ation

… jackson json handling

…onfig.java to ProTeGiOptimizer(Config) for clarity - add doc section about the ProTeGi optimizer - add new "protegi" config alias but keep previous "gradient" for compatibility

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/LazyChatModel.java

Copilot

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/ProTeGiOptimizer.java

Copilot · 2026-03-06T00:11:49Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/configuration/OptimizerConfiguration.java

 public record OptimizerConfiguration(
        @JsonUnwrapped EvaluationConfiguration evaluationConfiguration,
-        @JsonProperty("prompt_optimizer") ModuleConfiguration promptOptimizer,
-        @JsonProperty("metric") ModuleConfiguration metric)
+        @JsonProperty(PROMPT_OPTIMIZER_FIELD) ModuleConfiguration promptOptimizer,
+        @JsonProperty("metric") ModuleConfiguration metric,
+        @JsonProperty("evaluator") ModuleConfiguration evaluator)


Making evaluator a mandatory field forces all existing optimizer configs to include an evaluator section; otherwise Jackson deserialization will fail. If backward compatibility is desired, consider making this field @Nullable and defaulting Selector.createSelector(null) to MockSelector (as it already does), and guard evaluator.finalizeForSerialization() accordingly.

example-configs/gradient-optimizer-config.json

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/MockOptimizer.java

src/test/java/edu/kit/kastel/sdq/lissa/ratlr/e2e/Requirement2RequirementE2ETest.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/EvaluationResult.java

...u/kit/kastel/sdq/lissa/ratlr/promptoptimizer/promptselector/UpperConfidenceBoundBandits.java

docs/prompt-optimization.md

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/ProTeGiOptimizer.java

…roTeGiOptimizer.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/ProTeGiOptimizer.java

docs/prompt-optimization.md

Copilot · 2026-03-06T00:27:34Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/configuration/OptimizerConfiguration.java

        evaluationConfiguration.serializeAndDestroyConfiguration();
        promptOptimizer.finalizeForSerialization();
        metric.finalizeForSerialization();
+        evaluator.finalizeForSerialization();


serializeAndDestroyConfiguration() unconditionally calls evaluator.finalizeForSerialization(). If an optimization config omits the new evaluator section (e.g., older configs), evaluator will deserialize as null and this will throw a NullPointerException. Consider marking evaluator as @Nullable and null-checking here (defaulting to a mock selector), or enforce non-null by validation with a clearer error message.

Suggested change

evaluator.finalizeForSerialization();

if (evaluator != null) {

evaluator.finalizeForSerialization();

}

src/test/java/edu/kit/kastel/sdq/lissa/ratlr/ArchitectureTest.java

…iles for consistency - various additional small tweaks

sonarqubecloud · 2026-03-06T09:50:35Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/configuration/OptimizerConfiguration.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/samplestrategy/SamplerFactory.java

dfuchss · 2026-03-06T00:11:45Z

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/EvaluationResult.java

+     * Checks if the classification does not match the ground truth.
+     * @return false if the classification is equal to the ground truth, true otherwise
+     */
+    public boolean isIncorrect() {


try to flip this. typically you would ask for "isCorrect"

This function is only used in this negated form, flipping it would mean always using !isCorrect(). How about renaming it to isWrong() instead?

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/Statistics.java

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/configuration/OptimizerConfiguration.java

DanielDango added 5 commits January 22, 2026 12:50

feat: Add gradient optimizer based on ProTeGi by "Automatic Prompt Op…

dc7426a

…timization with "Gradient Descent" and Beam Search" of Pryzant et al. 2023

feat: implement caching for OpenAI chat models to optimize performance

afc5741

feat: return intermediate results during prompt optimization

2adc722

feat: encapsulate cache replacement strategies into enum

778d5f0

- add new redis cache replacement strategy

Merge remote-tracking branch 'upstream/feature/add-prompt-optimizatio…

aecfb43

…n-module' into feature/add-protegi-optimization

DanielDango requested a review from dfuchss as a code owner January 22, 2026 11:53

Copilot AI review requested due to automatic review settings January 22, 2026 11:53

Copilot started reviewing on behalf of DanielDango January 22, 2026 11:53 View session

DanielDango changed the title ~~Feature/add protegi optimization~~ Add ProTeGi Prompt Optimizer Jan 22, 2026

DanielDango mentioned this pull request Jan 22, 2026

Add prompt optimizers to LiSSA #44

Merged

Copilot AI reviewed Jan 22, 2026

View reviewed changes

dfuchss mentioned this pull request Jan 22, 2026

Import lissa from jamieklein (squashed): replace upstream lissa with … #37

Closed

DanielDango added 9 commits February 16, 2026 16:51

Revert "revert: remove evaluator from config which is to be added wit…

279d7c3

…h protegi" This reverts commit d5f758b.

Revert "revert: Evaluator module only required for future gradient op…

35afddd

…timizer" This reverts commit 83b5b7b

refactor: fix package hierarchy

bedf3f1

revert: remove CacheReplacementStrategy.java

14e5f71

chore: pull recent changes into ProTeGi implementation

23c2750

fix: rename logger

891081c

Merge branch 'feature/add-prompt-optimization-module' into feature/ad…

bded850

…d-protegi-optimization

dfuchss changed the base branch from feature/add-prompt-optimization-module to main February 19, 2026 13:08

dfuchss added 2 commits February 19, 2026 15:39

Merge remote-tracking branch 'origin/main' into feature/add-protegi-o…

678fae1

…ptimization

Spotless applied

8eabec5

dfuchss reviewed Feb 19, 2026

View reviewed changes

dfuchss and others added 4 commits February 19, 2026 21:32

Make chat language model provider lazy

6523e69

chore: update optimize methods to return lists of optimized prompts

05a5877

fix: add docs and address APO issues

6929d4b

refactor: clarify evaluator module and rename to Selector

904d9dd

- rename sampleAndEvaluate to selectAndEvaluate - add documentation - rework abstract base class as interface

Copilot AI review requested due to automatic review settings March 5, 2026 11:12

Copilot started reviewing on behalf of DanielDango March 5, 2026 11:13 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

DanielDango added 4 commits March 5, 2026 16:40

chore: standardize configuration parameter names and improve document…

fa8174a

…ation

refactor: replace maximum_iterations statistic regex replacement with…

5093db3

… jackson json handling

chore: extend configurable fields for the AutomaticPromptOptimizer

271efd2

refactor: rename AutomaticPromptOptimizer.java and GradientOptimizerC…

d333c3e

…onfig.java to ProTeGiOptimizer(Config) for clarity - add doc section about the ProTeGi optimizer - add new "protegi" config alias but keep previous "gradient" for compatibility

DanielDango requested review from Copilot and dfuchss March 5, 2026 17:28

Copilot AI reviewed Mar 5, 2026

View reviewed changes

DanielDango requested a review from Copilot March 5, 2026 17:58

Copilot AI reviewed Mar 5, 2026

View reviewed changes

dfuchss reviewed Mar 6, 2026

View reviewed changes

src/main/java/edu/kit/kastel/sdq/lissa/ratlr/classifier/LazyChatModel.java Outdated Show resolved Hide resolved

Apply suggestion from @dfuchss

5564254

Copilot AI review requested due to automatic review settings March 6, 2026 00:04

Copilot started reviewing on behalf of dfuchss March 6, 2026 00:05 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Update src/main/java/edu/kit/kastel/sdq/lissa/ratlr/promptoptimizer/P…

dc33542

…roTeGiOptimizer.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 6, 2026 00:21

Copilot started reviewing on behalf of dfuchss March 6, 2026 00:22 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

refactor: rename evaluator to selector in configuration and related f…

7424af6

…iles for consistency - various additional small tweaks

dfuchss reviewed Mar 6, 2026

View reviewed changes

Conversation

DanielDango commented Jan 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dfuchss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dfuchss Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

dfuchss Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

DanielDango Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!