CLI: only deploy and pull credentials being referenced in jobs

[midigofrank]

Right now, pull will fetch all credentials linked to the project and you might end up with a bloated file. Is it possible to only write/include only credentials linked to the jobs?

Similarly with deploy, all credentials listed will be pushed even though they're not being used in the jobs. Well, deploy is a consequence to pull, if we fix pull then we won't have the "extra" credentials in the first place.

deploy is where it hurts the most tbh, when you want to sync a project from app.openfn and the project has "extra" credentials, you end up having to cleanup the data first before deploying to the target project. Maybe a flag could help here? Like deploy --only-active-credentials ??

[josephjclark]

@midigofrank I hear you on this, but where I get stuck is I don't think this is the CLI's fault.

Seems to be the provisioner should be responsible for deciding which credentials to return on fetch.

I consider the credentials to be a Lightning concept. The CLI doesn't know what a credential is. The CLI just runs with state. Recently we've added some support to associated state with a credential UUID, but really this is just the CLI faking it. The CLI relies on Lightning giving it an accurate credentials array, and I don't think the CLI should maintain it. Because it's up to humans in the app to decide what credentials are associated with a project.

you end up having to cleanup the data first before deploying to the target project

Is this because errors are thrown, or is just about removing bloat? I'm not really sure why I understand why bloat is a problem, or even why some projects are in this state. Should those credentials not just be removed from the project?

If you're in a bind I can look into a temporary CLI workaround, but this feels like a hack - not a solution

[midigofrank]

Yes, the problem is an error will be thrown. You'll find a case where the project in app.openfn has extra test credentials associated with the project because collaborators are actively building in there. The target project on the other hand is lean, it only contains the credentials being used.

But I do agree with you, this is a lightning problem. I'm just wondering if maybe there are users who would want that whole list of credentials included, because then it becomes a lightning + cli issue