[Feature][EngineConn] Optimize concurrent engine EC creation to avoid resource waste #5318
Description
Search before asking
- I searched the issues and found no similar issues.
Linkis Component
linkis-computation-governance
What happened
English:
For concurrent engine plugins (such as Shell, JDBC), a single EC can execute multiple tasks (e.g., 10 tasks). However, when no EC is available for reuse, a new EC will be created.
Problem Scenario:
When a user submits 30 tasks at once and no EC is available for reuse, the system will start 30 ECs simultaneously.
Actual Requirement:
In fact, only 3 ECs are needed to meet the user's requirement of 30 concurrent tasks (each EC can handle 10 concurrent tasks).
中文：
对于并发引擎插件（如Shell、JDBC等），单个EC可以执行多个任务（比如10个）。但在没有EC可以复用的情况下会创建EC。
问题场景：
用户一次性提交30个任务，此时没有EC可以复用，系统会同时启动30个EC。
实际需求：
实际上只需要启动3个EC就能满足用户并发30的要求（每个EC可以处理10个并发任务）。
What you expected to happen
English:
The system should intelligently create the optimal number of ECs based on the engine's concurrency capability:
- Detect the single EC concurrency capability (e.g., JDBC EC supports 10 concurrent tasks)
- Calculate the optimal number of ECs based on total tasks and single EC concurrency:
Required ECs = ceil(Total Tasks / EC Concurrency) - Avoid excessive EC creation that wastes resources
Expected Behavior:
- When submitting 30 tasks, the system creates 3 ECs (30/10=3)
- Each EC handles 10 concurrent tasks
- Save resources and improve resource utilization
中文：
系统应该根据引擎的并发能力智能创建EC数量：
- 检测引擎的单EC并发能力（如JDBC EC支持10个并发）
- 根据任务总数和单EC并发能力计算最优EC数量：
需要EC数 = ceil(任务数 / 单EC并发数) - 避免过度创建EC造成资源浪费
期望行为：
- 提交30个任务时，系统创建3个EC（30/10=3）
- 每个EC处理10个并发任务
- 节约资源，提高资源利用率
How to reproduce
English:
- Configure Shell or JDBC engine with single EC concurrency set to 10
- Ensure there are no reusable EC instances
- Submit 30 tasks at once
- Observe that the system creates 30 ECs instead of 3
中文：
- 配置Shell或JDBC引擎，设置单EC并发数为10
- 确保当前没有可复用的EC实例
- 一次性提交30个任务
- 观察系统会创建30个EC而不是3个
Anything else
English:
Suggested Implementation:
Add intelligent EC creation logic in EngineConnManager:
- Get the concurrency configuration for the engine type
- Calculate the current task queue length
- Create ECs based on formula:
Required ECs = ceil(Task Count / Single EC Concurrency) - Implement concurrency control for EC creation to avoid instantaneous mass creation
Impact:
This issue is particularly evident in high-concurrency scenarios, causing:
- Resource waste (creating too many ECs)
- Increased startup time (startup time of 30 ECs >> startup time of 3 ECs)
- Potential resource shortage (if resource pool is limited)
中文：
建议实现方案：
在EngineConnManager中增加智能EC创建逻辑：
- 获取引擎类型的并发配置
- 计算当前任务队列长度
- 根据公式创建EC：
需要EC数 = ceil(任务数 / 单EC并发数) - 实现EC创建的并发控制，避免瞬时大量创建
影响范围：
这个问题在高并发场景下尤为明显，会造成：
- 资源浪费（创建过多EC）
- 启动时间增加（30个EC启动时间 >> 3个EC启动时间）
- 可能导致资源不足（如果资源池有限）
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Linkis Version
1.8.0
Java Version
OpenJDK 1.8.0