Few shot prompt enhancement #298
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request aims to enhance the skill extraction prompt by adding few-shot examples to improve the model's extraction accuracy. The changes rename the method from skill_extraction_prompt to fs_gold_skill_extraction_prompt and add three examples demonstrating different skill types: soft skills & communication, math & technical background, and core responsibilities.
Key Changes:
- Renamed method
skill_extraction_prompttofs_gold_skill_extraction_prompt - Added three few-shot examples to the prompt demonstrating skill extraction patterns
- Changed internal variable name from
prompttostandard_prompt
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Example 1 (Focus: Soft Skills & Communication) Input: "Strong verbal and written communication skills, with the ability to explain complex technical concepts clearly to both technical and non-technical audiences. Confident presenter, capable of articulating insights, results, and strategies to stakeholders." Output: ['Strong verbal and written communication skills', 'explain complex technical concepts', 'Confident presenter', 'capable of articulating insights'] | ||
|
|
||
| Example 2 (Focus: Math & Technical Background) Input: "Qualified candidates will have a strong mathematical background (statistics, linear algebra, calculus, probability, and optimization). Experience with deep learning, natural language processing, or application of large language models is preferred." Output: ['Strong mathematical background', 'statistics, linear algebra, calculus, probability, and optimization', 'deep learning', 'natural language processing', 'application of large language models'] | ||
|
|
||
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['Lead the research, design, implementation, and deployment of Machine Learning algorithms', 'Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite.', 'Contribute to the design and implementation of new features of the C3 AI Suite.'] |
There was a problem hiding this comment.
The output in this example contains full sentences that violate the formatting rule "Each skill string must be between 1 and 5 words" specified later in the prompt. For instance, 'Assist and enable C3 AI's federal customers to build their own applications on the C3 AI Suite.' is much longer than 5 words. The examples should be consistent with the formatting rules.
| Example 1 (Focus: Soft Skills & Communication) Input: "Strong verbal and written communication skills, with the ability to explain complex technical concepts clearly to both technical and non-technical audiences. Confident presenter, capable of articulating insights, results, and strategies to stakeholders." Output: ['Strong verbal and written communication skills', 'explain complex technical concepts', 'Confident presenter', 'capable of articulating insights'] | |
| Example 2 (Focus: Math & Technical Background) Input: "Qualified candidates will have a strong mathematical background (statistics, linear algebra, calculus, probability, and optimization). Experience with deep learning, natural language processing, or application of large language models is preferred." Output: ['Strong mathematical background', 'statistics, linear algebra, calculus, probability, and optimization', 'deep learning', 'natural language processing', 'application of large language models'] | |
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['Lead the research, design, implementation, and deployment of Machine Learning algorithms', 'Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite.', 'Contribute to the design and implementation of new features of the C3 AI Suite.'] | |
| Example 1 (Focus: Soft Skills & Communication) Input: "Strong verbal and written communication skills, with the ability to explain complex technical concepts clearly to both technical and non-technical audiences. Confident presenter, capable of articulating insights, results, and strategies to stakeholders." Output: ['verbal communication', 'written communication', 'technical explanation', 'stakeholder presentations'] | |
| Example 2 (Focus: Math & Technical Background) Input: "Qualified candidates will have a strong mathematical background (statistics, linear algebra, calculus, probability, and optimization). Experience with deep learning, natural language processing, or application of large language models is preferred." Output: ['mathematical background', 'statistical methods', 'linear algebra', 'deep learning', 'natural language processing'] | |
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['ML algorithm design', 'customer application development', 'feature design'] |
|
|
||
| Example 2 (Focus: Math & Technical Background) Input: "Qualified candidates will have a strong mathematical background (statistics, linear algebra, calculus, probability, and optimization). Experience with deep learning, natural language processing, or application of large language models is preferred." Output: ['Strong mathematical background', 'statistics, linear algebra, calculus, probability, and optimization', 'deep learning', 'natural language processing', 'application of large language models'] | ||
|
|
||
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['Lead the research, design, implementation, and deployment of Machine Learning algorithms', 'Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite.', 'Contribute to the design and implementation of new features of the C3 AI Suite.'] |
There was a problem hiding this comment.
The output list for Example 3 includes text that was not present in the input. The input says "Contribute to the design of new features" but the output shows "Contribute to the design and implementation of new features of the C3 AI Suite." This adds "and implementation" and "of the C3 AI Suite" which were not in the original text, contradicting the extraction instruction "Do not invent new skills or make assumptions beyond the provided text."
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['Lead the research, design, implementation, and deployment of Machine Learning algorithms', 'Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite.', 'Contribute to the design and implementation of new features of the C3 AI Suite.'] | |
| Example 3 (Focus: Core Responsibilities) Input: "Lead the research, design, implementation, and deployment of Machine Learning algorithms. Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite. Contribute to the design of new features." Output: ['Lead the research, design, implementation, and deployment of Machine Learning algorithms', 'Assist and enable C3 AI’s federal customers to build their own applications on the C3 AI Suite.', 'Contribute to the design of new features'] |
| def skill_extraction_prompt(self, cleaned_description): | ||
| prompt = f""" | ||
|
|
||
| def fs_gold_skill_extraction_prompt(cleaned_description): |
There was a problem hiding this comment.
The method is missing the 'self' parameter. This method appears to be part of the SkillExtractorRefactored class but is defined without 'self', which will cause a runtime error when calling this method as an instance method. The parameter should be 'self, cleaned_description' instead of just 'cleaned_description'.
| fs_gold_prompt = fs_gold_skill_extraction_prompt(cleaned_description) | ||
|
|
There was a problem hiding this comment.
This line has incorrect indentation and references an undefined variable 'cleaned_description'. It appears at module level (outside the class), but 'cleaned_description' is not defined in this scope. This line should either be removed or properly integrated into a method where 'cleaned_description' is available.
| fs_gold_prompt = fs_gold_skill_extraction_prompt(cleaned_description) |
|
https://colab.research.google.com/drive/1zJ6Q1ZJjh_J4NdxwpBdyQyoe13-ceWye#scrollTo=B1lbIaWlEuUW Updates are made based on the above notebook. |
No description provided.