Skip to content

Single-controller Realm backend #1629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
elliottslaughter wants to merge 49 commits into
base: master
Choose a base branch
from
Draft

Single-controller Realm backend #1629

elliottslaughter wants to merge 49 commits into from

Conversation

@elliottslaughter
Copy link
Contributor

@elliottslaughter elliottslaughter commented Feb 12, 2026 •
edited by wmdi
Loading

This PR implements a single-controller Realm execution backend.

Feature overview:

  • Single-controller (i.e., controller runs on one node)
  • One task per dynamic graph node
  • Most of the infrastructure for distributed execution is present, except:
    • Copies are not inserted, so data will not be moved between devices
    • Task arguments are passed by pointer rather than serialized, so this will not be safe to run on multiple nodes

This change is Reviewable

@elliottslaughter
Add make_dynamic_open_dataflow_graph_from_pcg.
6a74af0
@elliottslaughter
Empty skeleton of the realm-execution backend.
587e08e
@elliottslaughter
More Realm execution skeleton.
50f6ec6
@elliottslaughter
Stub creation.
984aae5
@elliottslaughter
More passes.
e9e1105
@elliottslaughter
Add Realm manager and test it.
90b92c7
@elliottslaughter
Do not expose raw runtime and properly wait in test.
ef92d6f
@elliottslaughter
Sketch more Realm manager APIs.
01e23cd
@elliottslaughter
Add controller functionality.
3e7d841
@elliottslaughter
Fix Realm tests.
b9a30a6
@elliottslaughter
Support passing closure arguments to controllers.
814e13f
@elliottslaughter
Move task IDs into Realm and assign IDs to remaining tasks.
3d0298c
@elliottslaughter
Avoid pulling in the entire invocation.
d702afe
@elliottslaughter
Conversion into Realm task IDs.
4fcde77
@elliottslaughter
Add a top-level PRealm switch.
e51b04e
@elliottslaughter
Some work on Realm task registry.
895de33
@elliottslaughter
Split out the Realm context.
09fde7d
@elliottslaughter
Switch to mapped PCG.
c5a0ea9
@elliottslaughter
Add shard expansion pass (and implement shard expansion pass).
a587e53
@elliottslaughter
Add instance field to dynamic graph, more task IDs.
62b49f7
@elliottslaughter
Fix filename.
ce403d4
@elliottslaughter
Some work in instance allocation and registry/manager.
a4183dd
@elliottslaughter
Instance allocation.
0274dd0
@elliottslaughter
Simplify dims and use constructors.
9d24b3d
@elliottslaughter
Refactor.
60989fe
@elliottslaughter
Sketch out device mapping.
8d46441
@elliottslaughter
Move instance backing to a separate map, remove realm from task-spec.
0dfa1a3
@elliottslaughter
Implement processor queries.
a4bc84e
@elliottslaughter
Enable PRealm.
02b71a8
@elliottslaughter
Move tasks to dedicated file, stub out device state init, shuffle dir…
b144d6d 
…ectories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@elliottslaughter