Skip to content

Poor performance when UploadFolder procedure when uploading folder with 5000 files #98

@nikhilv

Description

@nikhilv

To reproduce:

  1. Create a folder with 5000 files
  2. Create a project and procedure
  3. Create a subprocedure call invoking EC-S3:UploadFolder
  4. Specify directory from 1
  5. Run Procedure

EXPECTED RESULT:
Long pole should be communication to S3.

ACTUAL RESULT:
Long pole seems to be setting properties on the Flow Server.

Initial Triage:
When uploading a folder with 5000 files, it looks like we try to do 5000 serial setProperty requests to the FlowServer.

   //set the properties
    list.each {
        item ->
            def url = "https://" + bucketName + ".s3.amazonaws.com/" + item
            System.out.println(item + "  ==>  [" + url + "]")
            commander.setProperty(propResult + "/" + item, url)
    }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions