feat: devnet3 changes

README.md

@@ -82,6 +82,19 @@ Grafana is started with the two pre-provisioned dashboards from [leanMetrics](ht
 
 > **Note:** The `--metrics` flag only affects local deployments. When using Ansible deployment mode, this flag is ignored. Metrics ports are always exposed by clients regardless of this flag.
 
+### Aggregator Selection
+
+```sh
+# Let the system randomly select an aggregator (default behavior)
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis
+
+# Manually specify which node should be the aggregator
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --aggregator zeam_0
+
+# The aggregator selection is applied automatically and the isAggregator flag
+# is updated in validator-config.yaml before nodes are started
+```
+
 ## Args
 
 1. `NETWORK_DIR` is an env to specify the network directory. Should have a `genesis` directory with genesis config. A `data` folder will be created inside this `NETWORK_DIR` if not already there.
@@ -139,8 +152,14 @@ Grafana is started with the two pre-provisioned dashboards from [leanMetrics](ht
     - On Ctrl+C cleanup, the metrics stack is stopped automatically
 
     Note: Client metrics endpoints are always enabled regardless of this flag.
-12. `--checkpoint-sync-url` specifies the URL to fetch finalized checkpoint state from for checkpoint sync. Default: `https://leanpoint.leanroadmap.org/lean/v0/states/finalized`. Only used when `--restart-client` is specified.
-13. `--restart-client` comma-separated list of client node names (e.g., `zeam_0,ream_0`). When specified, those clients are stopped, their data cleared, and restarted using checkpoint sync. Genesis is skipped. Use with `--checkpoint-sync-url` to override the default URL.
+12. `--aggregator` specifies which node should act as the aggregator (1 aggregator per subnet).
+   - If not provided, one node will be randomly selected as the aggregator
+   - If provided, the specified node will be set as the aggregator
+   - The aggregator selection updates the `isAggregator` flag in `validator-config.yaml`
+   - Example: `--aggregator zeam_0` to make zeam_0 the aggregator
+   - Example: Without flag, a random node will be selected automatically
+13. `--checkpoint-sync-url` specifies the URL to fetch finalized checkpoint state from for checkpoint sync. Default: `https://leanpoint.leanroadmap.org/lean/v0/states/finalized`. Only used when `--restart-client` is specified.
+14. `--restart-client` comma-separated list of client node names (e.g., `zeam_0,ream_0`). When specified, those clients are stopped, their data cleared, and restarted using checkpoint sync. Genesis is skipped. Use with `--checkpoint-sync-url` to override the default URL.
 
 ### Checkpoint sync
 

TESTING_DEVNET3.md

@@ -0,0 +1,159 @@
+# Testing devnet3 Configuration
+
+## Setup
+
+The local-devnet has been configured to test devnet3 features with the local zeam image.
+
+### Changes Made:
+
+1. **zeam-cmd.sh**: Updated to use `0xpartha/zeam:local` image
+2. **local-devnet/validator-config.yaml**: Added commented attestation_committee_count parameter
+
+## devnet3 Features Being Tested
+
+### 1. Automatic Aggregator Selection
+- One node will be randomly selected as aggregator on startup
+- The `isAggregator` flag in validator-config.yaml will be automatically updated
+- Only the selected aggregator will receive `--is-aggregator` flag
+
+### 2. Optional Attestation Committee Count
+- Currently commented out (clients use hardcoded default)
+- Can be enabled by uncommenting: `attestation_committee_count: 1`
+- When set, all clients receive `--attestation-committee-count <value>` flag
+
+## Test Commands
+
+### Basic Test - Single Node (zeam_0)
+```bash
+# Run the test script
+./test-local-zeam.sh
+
+# Or manually:
+NETWORK_DIR=local-devnet ./spin-node.sh --node zeam_0 --generateGenesis --cleanData
+```
+
+### Test with Manual Aggregator Selection
+```bash
+# Specify zeam_0 as aggregator
+NETWORK_DIR=local-devnet ./spin-node.sh --node zeam_0 --generateGenesis --cleanData --aggregator zeam_0
+```
+
+### Test Multiple Nodes
+```bash
+# Run all nodes (zeam_0 will be randomly selected as aggregator)
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --cleanData
+
+# Run specific nodes with zeam_0 as aggregator
+NETWORK_DIR=local-devnet ./spin-node.sh --node "zeam_0 ream_0" --generateGenesis --cleanData --aggregator zeam_0
+```
+
+### Test with Attestation Committee Count Override
+```bash
+# 1. Uncomment attestation_committee_count in local-devnet/genesis/validator-config.yaml
+# 2. Set desired value (e.g., attestation_committee_count: 4)
+# 3. Run:
+NETWORK_DIR=local-devnet ./spin-node.sh --node zeam_0 --generateGenesis --cleanData
+```
+
+## Expected Behavior
+
+### Aggregator Selection
+1. Script displays: "Randomly selected aggregator: zeam_0 (index 0 out of 7 nodes)" OR "Using user-specified aggregator: zeam_0"
+2. Script updates validator-config.yaml: sets `isAggregator: true` for selected node
+3. parse-vc.sh output shows: "Is Aggregator: true" for aggregator, "false" for others
+4. zeam command includes `--is-aggregator` flag for aggregator only
+
+### Attestation Committee Count
+**When NOT set (default):**
+- parse-vc.sh does NOT display "Attestation Committee Count"
+- zeam command does NOT include `--attestation-committee-count` flag
+- Client uses its hardcoded default
+
+**When set (e.g., to 4):**
+- parse-vc.sh displays: "Attestation Committee Count: 4"
+- zeam command includes: `--attestation-committee-count 4`
+- Client uses the specified value
+
+## Verification
+
+### Check Docker Container
+```bash
+# Inspect the running container
+docker inspect zeam_0
+
+# Check container logs
+docker logs zeam_0
+
+# Verify command-line arguments
+docker inspect zeam_0 | grep -A20 Args
+```
+
+### Check Configuration
+```bash
+# Verify aggregator selection in validator-config.yaml
+yq eval '.validators[] | select(.name == "zeam_0") | .isAggregator' local-devnet/genesis/validator-config.yaml
+
+# Check all aggregators
+yq eval '.validators[] | select(.isAggregator == true) | .name' local-devnet/genesis/validator-config.yaml
+```
+
+### Monitor Node Output
+```bash
+# Watch zeam_0 logs in real-time
+docker logs -f zeam_0
+
+# Check for aggregator-related messages
+docker logs zeam_0 2>&1 | grep -i aggregat
+```
+
+## Cleanup
+
+```bash
+# Stop and remove zeam_0 container
+docker rm -f zeam_0
+
+# Stop all nodes
+NETWORK_DIR=local-devnet ./spin-node.sh --node all --stop
+
+# Clean data directories
+rm -rf local-devnet/data/*
+```
+
+## Troubleshooting
+
+### Image Not Found
+If you get "image not found" error:
+```bash
+# Check if image exists
+docker images | grep zeam
+
+# Pull/build the image if needed
+# (build instructions depend on your zeam setup)
+```
+
+### Port Conflicts
+If ports are already in use:
+```bash
+# Check what's using the port
+lsof -i :8081  # zeam_0 metrics port
+lsof -i :9001  # zeam_0 QUIC port
+
+# Kill conflicting processes or change ports in validator-config.yaml
+```
+
+### Genesis Generation Fails
+```bash
+# Ensure yq is installed
+brew install yq  # macOS
+# or follow: https://github.com/mikefarah/yq#install
+
+# Check validator-config.yaml syntax
+yq eval . local-devnet/genesis/validator-config.yaml
+```
+
+## Notes
+
+- The `0xpartha/zeam:local` image should have the latest devnet3 changes
+- All devnet3 features (aggregator selection, optional attestation_committee_count) are enabled
+- The configuration is set up for local testing with 127.0.0.1 IPs
+- Hash-sig keys will be generated automatically on first run with `--generateGenesis`

ansible/roles/ethlambda/tasks/main.yml

@@ -38,12 +38,14 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - isAggregator
   when: node_name is defined
 
-- name: Set node ports
+- name: Set node ports and aggregator flag
   set_fact:
     ethlambda_quic_port: "{{ ethlambda_node_config.results[0].stdout }}"
     ethlambda_metrics_port: "{{ ethlambda_node_config.results[1].stdout }}"
+    ethlambda_is_aggregator: "{{ 'true' if (ethlambda_node_config.results[2].stdout | default('') | trim) == 'true' else 'false' }}"
   when: ethlambda_node_config is defined
 
 - name: Ensure node key file exists
@@ -97,6 +99,7 @@
         --node-key /config/{{ node_name }}.key
         --metrics-address 0.0.0.0
         --metrics-port {{ ethlambda_metrics_port }}
+        {{ '--is-aggregator' if (ethlambda_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: ethlambda_container
       changed_when: ethlambda_container.rc == 0

ansible/roles/grandine/tasks/main.yml

@@ -38,12 +38,16 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - privkey
+    - isAggregator
   when: node_name is defined
 
-- name: Set node ports
+- name: Set node ports and aggregator flag
   set_fact:
     grandine_quic_port: "{{ grandine_node_config.results[0].stdout }}"
     grandine_metrics_port: "{{ grandine_node_config.results[1].stdout }}"
+    grandine_privkey: "{{ grandine_node_config.results[2].stdout }}"
+    grandine_is_aggregator: "{{ 'true' if (grandine_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
   when: grandine_node_config is defined
 
 - name: Ensure node key file exists
@@ -102,6 +106,7 @@
         --metrics
         --http-address 0.0.0.0
         --http-port {{ grandine_metrics_port }}
+        {{ '--is-aggregator' if (grandine_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: grandine_container
       changed_when: grandine_container.rc == 0

ansible/roles/lantern/defaults/main.yml

@@ -3,6 +3,6 @@
 # Note: These are fallback defaults. Actual values are extracted from client-cmds/lantern-cmd.sh
 # in the tasks/main.yml file. These defaults are used if extraction fails.
 
-lantern_docker_image: "piertwo/lantern:latest"
+lantern_docker_image: "piertwo/lantern:v0.0.3"
 deployment_mode: docker  # docker or binary
 

ansible/roles/lantern/tasks/main.yml

@@ -2,16 +2,9 @@
 # Lantern role: Deploy and manage Lantern nodes
 # Converts client-cmds/lantern-cmd.sh logic to Ansible tasks
 
-- name: Extract docker image from client-cmd.sh
-  shell: |
-    # Extract the docker image from LANTERN_IMAGE line
-    # playbook_dir points to ansible/playbooks, go up two levels to reach project root
-    project_root="$(cd '{{ playbook_dir }}/../..' && pwd)"
-    grep -E '^LANTERN_IMAGE=' "$project_root/client-cmds/lantern-cmd.sh" | head -1 | sed -E 's/.*LANTERN_IMAGE="([^"]+)".*/\1/'
-  register: lantern_docker_image_raw
-  changed_when: false
-  delegate_to: localhost
-  run_once: true
+- name: Set lantern docker image
+  set_fact:
+    lantern_docker_image: "piertwo/lantern:v0.0.3"
 
 - name: Extract deployment mode from client-cmd.sh
   shell: |
@@ -23,9 +16,8 @@
   delegate_to: localhost
   run_once: true
 
-- name: Set docker image and deployment mode from client-cmd.sh
+- name: Set deployment mode from client-cmd.sh
   set_fact:
-    lantern_docker_image: "{{ lantern_docker_image_raw.stdout | trim | default('piertwo/lantern:latest') }}"
     deployment_mode: "{{ lantern_deployment_mode_raw.stdout | trim | default('docker') }}"
 
 - name: Extract node configuration from validator-config.yaml
@@ -37,12 +29,16 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - privkey
+    - isAggregator
   when: node_name is defined
 
-- name: Set node ports
+- name: Set node ports and aggregator flag
   set_fact:
     lantern_quic_port: "{{ lantern_node_config.results[0].stdout }}"
     lantern_metrics_port: "{{ lantern_node_config.results[1].stdout }}"
+    lantern_privkey: "{{ lantern_node_config.results[2].stdout }}"
+    lantern_is_aggregator: "{{ 'true' if (lantern_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
   when: lantern_node_config is defined
 
 - name: Ensure node key file exists
@@ -96,6 +92,7 @@
         --metrics-port {{ lantern_metrics_port }}
         --http-port 5055
         --hash-sig-key-dir /config/hash-sig-keys
+        {{ '--is-aggregator' if (lantern_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: lantern_container
       changed_when: lantern_container.rc == 0

ansible/roles/lighthouse/tasks/main.yml

@@ -37,12 +37,16 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - privkey
+    - isAggregator
   when: node_name is defined
 
-- name: Set node ports
+- name: Set node ports and aggregator flag
   set_fact:
     lighthouse_quic_port: "{{ lighthouse_node_config.results[0].stdout }}"
     lighthouse_metrics_port: "{{ lighthouse_node_config.results[1].stdout }}"
+    lighthouse_privkey: "{{ lighthouse_node_config.results[2].stdout }}"
+    lighthouse_is_aggregator: "{{ 'true' if (lighthouse_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
   when: lighthouse_node_config is defined
 
 - name: Ensure node key file exists
@@ -100,6 +104,7 @@
         --metrics
         --metrics-address 0.0.0.0
         --metrics-port {{ lighthouse_metrics_port }}
+        {{ '--is-aggregator' if (lighthouse_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: lighthouse_container
       changed_when: lighthouse_container.rc == 0

ansible/roles/qlean/tasks/main.yml

@@ -4,10 +4,9 @@
 
 - name: Extract docker image from client-cmd.sh
   shell: |
-    # Extract the first word (docker image) from node_docker line
-    # playbook_dir points to ansible/playbooks, go up two levels to reach project root
+    # Extract the docker image from QLEAN_IMAGE line (may be indented inside if/else)
     project_root="$(cd '{{ playbook_dir }}/../..' && pwd)"
-    grep -E '^node_docker=' "$project_root/client-cmds/qlean-cmd.sh" | head -1 | sed -E 's/.*node_docker="([^ "]+).*/\1/'
+    grep -E '^\s*QLEAN_IMAGE=' "$project_root/client-cmds/qlean-cmd.sh" | head -1 | sed -E 's/.*QLEAN_IMAGE="([^"]+)".*/\1/'
   register: qlean_docker_image_raw
   changed_when: false
   delegate_to: localhost
@@ -37,12 +36,16 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - privkey
+    - isAggregator
   when: node_name is defined
 
-- name: Set node ports
+- name: Set node ports and aggregator flag
   set_fact:
     qlean_quic_port: "{{ qlean_node_config.results[0].stdout }}"
     qlean_metrics_port: "{{ qlean_node_config.results[1].stdout }}"
+    qlean_privkey: "{{ qlean_node_config.results[2].stdout }}"
+    qlean_is_aggregator: "{{ 'true' if (qlean_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
   when: qlean_node_config is defined
 
 - name: Extract validator index from validators.yaml
@@ -108,7 +111,11 @@
         --node-id {{ node_name }}
         --node-key /config/{{ node_name }}.key
         --listen-addr /ip4/0.0.0.0/udp/{{ qlean_quic_port }}/quic-v1
-        --prometheus-port {{ qlean_metrics_port }}
+        --metrics-host 0.0.0.0
+        --metrics-port {{ qlean_metrics_port }}
+        --api-host 0.0.0.0
+        --api-port 5053
+        {{ '--is-aggregator' if (qlean_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
         -ldebug
       register: qlean_container