Set up BYOC on AWS

Run Tensorlake sandboxes in your AWS account, with executors on Amazon EC2 and either a Tensorlake-run or a self-hosted Sandbox Proxy.

Bring your own cloud (BYOC) lets your team run Tensorlake sandboxes on compute that you own. Your code, sandbox storage, and executor capacity stay in your AWS account. Your data never leaves your AWS account, and Tensorlake has no access to data in your AWS account. Tensorlake Cloud remains the control plane that schedules work and gives your team one API for managing sandboxes.

Sandboxes always run on executors in your account.

Overview

BYOC on AWS has two deployment modes. Both run sandboxes on executors in your own AWS account and keep your project bucket in your account. They differ in who runs the Sandbox Proxy, the component that terminates sandbox traffic and forwards it to an executor.

Compute in your network

Compute and ingress in your network

Choose a deployment mode

	Compute in your network	Compute and ingress in your network
You run	Executors	Executors and the Sandbox Proxy
Tensorlake runs	The Sandbox Proxy and the control plane	The control plane
You also need	A VPC peering connection to Tensorlake	An EKS cluster, an NLB, an ACM certificate, and a public DNS zone
Public endpoint	Tensorlake gives your project a dedicated endpoint	You own the endpoint and its domain

Before you begin

Tensorlake requirements

You need a Tensorlake project with Self-hosted Compute enabled. You also need permission to change the project settings and create a project-admin API key. If the Self-hosted Compute page is not available, contact Tensorlake.

The same project-admin API key can prepare the Sandbox Proxy and enroll the first executor. Treat the key as a secret. Do not save it in an EC2 user-data script.

AWS account requirements

Prepare these resources in the AWS Region where you want to run sandboxes. Both modes need:

• A VPC.
• An S3 bucket for runtime data, built images, and durable sandbox snapshots.
• An EC2 instance profile that can use the project bucket.
• Capacity for at least one supported bare-metal EC2 instance: i7i.metal-24xl or i7i.metal-48xl.

Compute and ingress in your network also needs:

• An EKS cluster.
• The AWS Load Balancer Controller, version 2.7.2 or later.
• An IAM role that gives the controller permission to create an AWS Network Load Balancer (NLB), target groups, and security-group rules.
• A public DNS zone for the sandbox domain.
• An AWS Certificate Manager (ACM) certificate for the sandbox domain.

Sandbox base domain

If you run the ingress, choose the base domain before you create certificates or manifests. Every sandbox receives a subdomain below this base domain. For example, a base domain of sandboxes.customer.example produces URLs such as https://<sandbox-id>.sandboxes.customer.example.

The public endpoint must be an HTTPS origin. Do not add a port or path.

Tensorlake provides the domain in the compute-only mode.

1. Prepare project storage

Use a dedicated bucket for the project. It is the durable home for sandbox state, and executors write built sandbox images and snapshots into it. Keeping the bucket in the executor Region reduces latency and AWS data transfer costs.

Block public access to the bucket. Enable encryption. If you use an AWS Key Management Service (AWS KMS) customer-managed key, give the executor instance role permission to use that key.

The executor instance role needs these S3 permissions for the bucket:

Scope	Required actions
Bucket	s3:GetBucketLocation, s3:ListBucket, s3:ListBucketMultipartUploads
Objects in the bucket	s3:GetObject, s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, s3:ListMultipartUploadParts

The enrollment preflight writes, reads, and deletes a small object under each derived storage path. This test stops the installation before a sandbox starts if the instance role cannot use the storage.

Attach the role through an EC2 instance profile. Do not put static AWS access keys on the host. The Sandbox Proxy does not need this role or any other S3 permission.

Open the project settings in Tensorlake Cloud. Find Node configuration and set the bucket URI:

s3://customer-tensorlake

Save the node configuration before you enroll an executor. Tensorlake derives the runtime object, image-build, and snapshot paths from the bucket URI.

2. Prepare your network

Subnet layout

If using VPC CIDR 10.90.0.0/16, you can set up this subnet layout, one subnet per Availability Zone:

Subnet	Ranges	Purpose	Needed by
Public	10.90.0.0/24, 10.90.1.0/24, 10.90.2.0/24	NLB	Ingress in your network
Private	10.90.16.0/20, 10.90.32.0/20, 10.90.48.0/20	Kubernetes nodes and pods	Ingress in your network
Executor	10.90.64.0/24, 10.90.65.0/24, 10.90.66.0/24	Bare-metal EC2 instances	Both modes

If you peer with Tensorlake, use the 10.90.0.0/16 range. Tensorlake standardizes on it so that every peered deployment shares one network layout. Give each executor subnet its own route table, because you add return routes to them later.

If you run the ingress, tag all three public subnets with kubernetes.io/role/elb=1 so the AWS Load Balancer Controller can select them without relying on route-table discovery. See the Amazon EKS NLB requirements.

Outbound access

Allow outbound TCP 443 from the executor hosts, and from the EKS nodes if you run the ingress, to Tensorlake services and required AWS services. The EKS nodes must pull the public Sandbox Proxy image from public.ecr.aws. The executor must reach EC2 instance metadata through IMDSv2 and must reach the configured S3 bucket.

Never expose executor ports to the internet in either mode. mTLS means mutual Transport Layer Security: both endpoints use certificates to verify each other.

3. Connect your deployment mode

Complete the section for the mode you chose. Do not combine them.

Connect compute in your network

Send your account details

Give Tensorlake these values through your support channel:

Value	Example
AWS account ID	111111111111
AWS Region	us-east-1
VPC ID	vpc-0123456789abcdef0
VPC address range	10.90.0.0/16
Executor subnet ranges	10.90.64.0/24, 10.90.65.0/24, 10.90.66.0/24

The executor subnet ranges are the destinations Tensorlake routes to. List the exact subnets, not the whole VPC range.

Receive the onboarding packet

Tensorlake creates the peering connection from its side and sends you a packet with these fields:

Field	What it is
peering_connection_id	The connection to accept
requester_account_id	The Tensorlake AWS account
requester_vpc_id	The Tensorlake VPC
provider_node_cidrs	One /27 per Availability Zone, so three ranges. These are the only sources the proxy uses.
destination_cidrs	The executor subnets you sent, echoed back
destination_ports	TCP 5565 and TCP 8095-8097

Verify the account, VPC, ranges, and ports before you accept anything.

Accept the peering connection

aws ec2 accept-vpc-peering-connection \
  --region "$AWS_REGION" \
  --vpc-peering-connection-id "$PEERING_CONNECTION_ID"

Confirm the status:

aws ec2 describe-vpc-peering-connections \
  --region "$AWS_REGION" \
  --vpc-peering-connection-ids "$PEERING_CONNECTION_ID" \
  --query 'VpcPeeringConnections[0].Status.Code' \
  --output text

The status must be active.

Route return traffic

Add a route in all three executor subnet route tables for each provider_node_cidrs value, with the peering connection as the target. Traffic that arrives from the proxy has no return path until you do.

Allow the proxy into the executor security group

Allow each provider_node_cidrs value into every executor security group on TCP 5565 and TCP 8095-8097.

Allow the /27 node ranges only. Do not allow a pod range, and do not allow the whole Tensorlake VPC range. If restrictive network ACLs protect the executor subnets, allow both request and return traffic there as well.

Connect compute and ingress in your network

Create the load balancer security group

The public NLB terminates TLS and sends traffic to the Sandbox Proxy pods.

Direction	Port	Source or target	Purpose
Inbound	443	Internet or approved client networks	HTTPS, WebSocket, and CLI SSH
Inbound	22	Approved SSH client networks	Optional native SSH
Outbound	9080	Sandbox Proxy pod security group	Proxy HTTP traffic
Outbound	2222	Sandbox Proxy pod security group	Proxy SSH traffic

The generated Service asks the AWS Load Balancer Controller to manage the matching backend security-group rules. The controller IAM role must include permission to add and remove those rules.

Allow the proxy into the executor security group

Allow traffic from the Sandbox Proxy pod network or security group to the executor security group:

Port	Purpose
8096	Native SSH traffic to a sandbox
8097	Dataplane API traffic over mutual TLS (mTLS)
5565	Raw sandbox TCP traffic over mTLS

Create the public certificate

Request an ACM certificate in the same AWS account and Region as the NLB. Add both names to the certificate:

sandboxes.customer.example
*.sandboxes.customer.example

Create the DNS validation records. Wait until ACM shows the certificate status as ISSUED.

Prepare the Sandbox Proxy

The preparation script creates the customer-specific mTLS certificate, two SSH key pairs, and a local Kustomize overlay. Kustomize is a Kubernetes manifest customization tool. The script creates files only in the output directory. It does not change your cluster or AWS account.

Download and run the script:

curl -fsSL \
  https://cloud.tensorlake.ai/byoc/sandbox-proxy/prepare.sh \
  --output prepare.sh

chmod +x prepare.sh

./prepare.sh \
  --api-key <tensorlake-api-key> \
  --endpoint https://sandboxes.customer.example \
  --namespace tensorlake-sandbox-proxy \
  --output-dir ./tensorlake-sandbox-proxy \
  --enable-aws-lb \
  --aws-lb-certificate-arn '<acm-certificate-arn>' \
  --aws-lb-security-group-id '<nlb-security-group-id>'

Keep the output directory secure. It contains private mTLS and SSH keys. Do not commit it to a source repository.

Set the SSH proxy public key

In the project settings in Tensorlake Cloud, set SSH proxy public key to the content of this generated file:

./tensorlake-sandbox-proxy/ssh/outbound.key.pub

The Sandbox Proxy uses the matching private key when it connects to a sandbox SSH server. Leave this setting empty only if you want to disable SSH for the project.

Tensorlake owns this key in the compute-only mode, so it is set only when you run the ingress yourself.

Deploy the Sandbox Proxy

Confirm the active Kubernetes context:

kubectl config current-context

Review the generated manifests and run a server-side dry run:

kubectl kustomize ./tensorlake-sandbox-proxy

kubectl apply \
  --server-side \
  --dry-run=server \
  -k ./tensorlake-sandbox-proxy

Add a Deployment patch if the cluster requires a node selector or toleration. Then apply the resources:

kubectl apply -k ./tensorlake-sandbox-proxy

kubectl \
  --namespace tensorlake-sandbox-proxy \
  rollout status deployment/sandbox-proxy

Wait for the controller to publish the NLB hostname:

kubectl \
  --namespace tensorlake-sandbox-proxy \
  get service sandbox-proxy-public

Confirm that both NLB target groups are healthy. The listener on port 443 uses pod port 9080. The listener on port 22 uses pod port 2222.

Create DNS records and activate the endpoint

Create two public DNS alias or CNAME records. Point both records to the NLB hostname:

sandboxes.customer.example
*.sandboxes.customer.example

You can use Amazon Route 53 or another public DNS provider.

Open the project settings in Tensorlake Cloud. Set Sandbox ingress endpoint to this value:

https://sandboxes.customer.example

Save the setting. Verify the public health route:

curl -fsS https://sandboxes.customer.example/health

Verify that the certificate also covers sandbox subdomains:

openssl s_client \
  -connect sandboxes.customer.example:443 \
  -servername test.sandboxes.customer.example \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -checkhost test.sandboxes.customer.example

The command prints Hostname test.sandboxes.customer.example does match certificate.

4. Launch the first executor

Launch a bare-metal EC2 instance with Ubuntu 24.04. Attach the instance profile from the storage step. The host must expose /dev/kvm because Tensorlake uses hardware virtualization to isolate sandboxes.

The instance must also have local NVMe instance storage. Enrollment formats those disks and mounts them at /mnt/data, which holds container images, sandbox images, and the local snapshot cache. Amazon EBS volumes are never used for this, so an instance type without local NVMe cannot run sandboxes.

These instance types meet both requirements:

i7i.metal-24xl
i7i.metal-48xl

Place the instance in an executor subnet that can receive Sandbox Proxy traffic. Enable IMDSv2. Give the root volume 50 GB so it holds the operating system and the downloaded components.

5. Enroll the executor

In Tensorlake Cloud, open Self-hosted Compute, select Add node, and copy the bootstrap command. It has this form:

curl -fsSL https://cloud.tensorlake.ai/byoc/bootstrap.sh \
  | sudo bash -s -- --api-key <tensorlake-api-key>

Run the command on the EC2 instance. The bootstrap process checks the host and S3 access. It then creates the executor identity, installs the provisioner, and starts the executor services.

The project-admin API key is used only during enrollment. The installed provisioner uses its own mTLS identity after enrollment. It continues to check Tensorlake for approved configuration and software updates.

Return to Self-hosted Compute. Confirm that the new node appears and becomes ready before you create a sandbox.

6. Verify the complete path

Install the Tensorlake CLI if the tl command is not available:

curl -fsSL https://tensorlake.ai/install | sh

Sign in and select the BYOC project:

tl login
tl init

Create a sandbox. Use its ID in the remaining commands:

tl sbx create
tl sbx exec <sandbox-id> sh -lc 'printf BYOC_OK'
tl sbx ssh <sandbox-id>

Every sandbox gets a subdomain of the project endpoint. If you run the ingress, that is your own base domain:

https://<sandbox-id>.sandboxes.customer.example

Test native SSH if port 22 is enabled:

tl sbx ssh keys add ~/.ssh/id_ed25519.pub
ssh <sandbox-id>@<sandbox-id>.sandboxes.customer.example

Create a snapshot and confirm that the executor writes data into the bucket:

tl sbx snapshot <sandbox-id>

Add more executors

Each executor needs its own EC2 instance and enrollment identity. Launch another supported host with the same network and instance-profile configuration. Then run a new command from Self-hosted Compute → Add node.

The project storage and Sandbox Proxy settings apply to all executors in the project. You do not need another proxy deployment or peering connection for each host.

Updates and key rotation

The provisioner runs as a service on every enrolled executor. It checks Tensorlake for new approved profiles and applies complete updates. A failed update keeps the last valid revision active.

If you run the ingress, rotate the Sandbox Proxy mTLS and SSH keys by running prepare.sh again with a new output directory. Review and apply the new resources. The certificate fingerprint starts a rolling pod update. Then replace SSH proxy public key with the new outbound.key.pub value.

Keep the previous output directory in secure storage until the rollout and end-to-end test pass.

Troubleshooting

Both modes:

Symptom	Check
The executor enrollment stops during host checks	Confirm Ubuntu 24.04, root access, IMDSv2, and /dev/kvm.
The storage preflight fails	Check the EC2 instance profile, S3 actions, bucket policy, and AWS KMS permissions.
A sandbox remains pending	Confirm that the executor is ready and belongs to the same Tensorlake project.

Compute in your network:

Symptom	Check
The peering connection stays pending-acceptance	Accept it in the Region and account named in the packet.
Sandboxes never become ready	Confirm the return routes in every executor subnet route table, not only the first.
Traffic arrives but nothing answers	Confirm the executor security group allows every provider_node_cidrs value on TCP 5565 and TCP 8095-8097.
Some Availability Zones work and others do not	The proxy uses one node range per zone. Confirm all of them appear in your routes and security groups.
The executor registers with a public address	Confirm the instance takes its address from instance metadata and sits in an executor subnet you sent.

Compute and ingress in your network:

Symptom	Check
The NLB is not created	Check the AWS Load Balancer Controller, its IAM role, and public subnet discovery.
An NLB target is unhealthy	Check pod readiness and traffic from the NLB security group to pod port 9080.
The proxy cannot reach a sandbox	Check private routes and executor security-group rules for ports 8096, 8097, and 5565.
HTTPS works but native SSH fails	Check the NLB listener on port 22, its target group, and both SSH security-group paths.