Documentation

Deploy Data Convoy workers in your AWS account.

This guide is for customers who subscribe to Data Convoy via AWS Marketplace. The Data Convoy web app runs on our infrastructure; you stand up one thing in your own AWS account: a worker host that moves your data and, on request, answers a few small AWS questions on the app's behalf. It's scripted as CloudFormation — you should not need to hand-author IAM. One naming note: the product is Data Convoy, while the software, binaries, and stack resources are named strata-* — they refer to the same thing.

What runs where

The worker is the only thing in your account with AWS credentials. It does the bulk copy, restore, and delete work, and it also answers the app's smaller requests — listing a prefix for the tree picker, running a connection test, reading CloudWatch bucket-size metrics, log housekeeping — over the same job-claim protocol it already uses, polling the app over HTTPS. The web app itself never holds or assumes any role in your account: there is nothing to trust, no ExternalId to protect, and no role ARN to paste anywhere.

Prerequisites

Have these in place before you launch the worker stack:

An AWS account

With your production and archive S3 buckets in it.

CloudFormation permissions

Permission to create IAM roles and EC2/Auto Scaling resources via CloudFormation.

Outbound network path

A VPC with subnets that have outbound internet access (NAT gateway or public subnets) so the worker can reach the app over HTTPS.

Set up in four steps

From subscription to a verified first job:

Create a team and set your buckets

Sign in to the web app, create or open a team, and set its production and archive bucket names in the team settings page.

Issue a worker token and launch the stack

In the team's Workers section, issue a worker token (shown once — copy it). From the AWS connection section, click Launch workers to open the pre-filled CloudFormation stack, then supply the token, the worker binary URL from your onboarding email, your VPC and subnets, and the instance type and worker count. Each instance boots, downloads the worker binary, and runs it as a systemd service (strata-worker).

Test the connection

Click Test connection in the AWS connection section. This queues a small job for a worker to pick up and reports ✓ Connected or the exact error. Green means the tree picker, restores, and log viewer can all reach your data.

Verify end to end

Submit a dry-run archive job for a small prefix from the Archive page. Within a few seconds a worker should claim it and the job should report counts without moving anything. You're live.

Wiring IAM by hand

The CloudFormation stack creates the worker's instance role for you, granting S3 and read-only CloudWatch metrics only — it never touches Secrets Manager or any key. If you wire IAM yourself instead, attach this policy to the worker's instance role, substituting the bucket-name placeholders:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBuckets",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": [
        "arn:aws:s3:::PROD-BUCKET",
        "arn:aws:s3:::ARCHIVE-BUCKET",
        "arn:aws:s3:::LOG-BUCKET"
      ]
    },
    {
      "Sid": "ObjectRW",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:RestoreObject"
      ],
      "Resource": [
        "arn:aws:s3:::PROD-BUCKET/*",
        "arn:aws:s3:::ARCHIVE-BUCKET/*",
        "arn:aws:s3:::LOG-BUCKET/*"
      ]
    },
    {
      "Sid": "BucketSizeMetrics",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    }
  ]
}

Scaling and teardown

Throughput scales with worker count: edit the worker stack's WorkerCount (or the Auto Scaling group's desired capacity) to add hosts — workers coordinate through the app's job leases, so it's safe to run several — or set it to 0 to pause without deleting the stack. To tear down, delete the worker stack: that stops and removes all worker hosts and revokes the app's ability to reach your buckets. There's only the one stack, so this is the entire teardown.

Troubleshooting

The failure modes we see, and what each one means:

Test connection or tree picker spins forever

These wait on a worker to claim a small job; if no worker host is running (or none is assigned to this team), they'll never resolve. Confirm at least one worker is online in the team's Workers section and assigned to this team.

Test connection fails with AccessDenied

The worker's instance role is missing a bucket permission. Confirm the worker stack's bucket parameters match the team's buckets, and that hand-wired IAM uses the current policy above (older versions predate the CloudWatch grant needed for bucket-size metrics).

Worker never claims jobs

Check the instance can reach the app URL over HTTPS (subnet has outbound internet) and that the worker token is valid — re-issue and update the stack if unsure. On the host: journalctl -u strata-worker.

Archive or restore fails with S3 AccessDenied

The worker instance role is missing a bucket permission; confirm the worker stack's bucket parameters match the team's buckets.

Your buckets. Your workers. Your control.

Make the next archive reversible.