> ## Documentation Index > Fetch the complete documentation index at: https://docs.tensorlake.ai/llms.txt > Use this file to discover all available pages before exploring further. # Set up BYOC on AWS > Run Tensorlake sandboxes in your AWS account, with executors on Amazon EC2 and either a Tensorlake-run or a self-hosted Sandbox Proxy. Bring your own cloud (BYOC) lets your team run Tensorlake sandboxes on compute that you own. Your code, sandbox storage, and executor capacity stay in your AWS account. Your data never leaves your AWS account, and Tensorlake has no access to data in your AWS account. Tensorlake Cloud remains the control plane that schedules work and gives your team one API for managing sandboxes. Sandboxes always run on executors in your account. ## Overview BYOC on AWS has two deployment modes. Both run sandboxes on executors in your own AWS account and keep your project bucket in your account. They differ in who runs the Sandbox Proxy, the component that terminates sandbox traffic and forwards it to an executor. ### Compute in your network Users reach a Sandbox Proxy that Tensorlake runs behind a load balancer in a Tensorlake AWS account. The proxy reaches executors in your AWS account across a VPC peering connection on private ports 5565 and 8095 to 8097. Executors write runtime data and snapshots to your Amazon S3 bucket, and both the proxy and the executors open outbound HTTPS connections to Tensorlake Cloud. Users reach a Sandbox Proxy that Tensorlake runs behind a load balancer in a Tensorlake AWS account. The proxy reaches executors in your AWS account across a VPC peering connection on private ports 5565 and 8095 to 8097. Executors write runtime data and snapshots to your Amazon S3 bucket, and both the proxy and the executors open outbound HTTPS connections to Tensorlake Cloud. ### Compute and ingress in your network Users reach an AWS Network Load Balancer in your AWS account. The load balancer sends traffic to the Sandbox Proxy on Amazon EKS, which reaches executors on bare-metal Amazon EC2 over the private VPC. Executors write runtime data and snapshots to Amazon S3. The Sandbox Proxy and the executors both open outbound HTTPS connections to Tensorlake Cloud. Users reach an AWS Network Load Balancer in your AWS account. The load balancer sends traffic to the Sandbox Proxy on Amazon EKS, which reaches executors on bare-metal Amazon EC2 over the private VPC. Executors write runtime data and snapshots to Amazon S3. The Sandbox Proxy and the executors both open outbound HTTPS connections to Tensorlake Cloud. ### Choose a deployment mode | | Compute in your network | Compute and ingress in your network | | --------------- | -------------------------------------------------- | ----------------------------------------------------------------- | | You run | Executors | Executors and the Sandbox Proxy | | Tensorlake runs | The Sandbox Proxy and the control plane | The control plane | | You also need | A VPC peering connection to Tensorlake | An EKS cluster, an NLB, an ACM certificate, and a public DNS zone | | Public endpoint | Tensorlake gives your project a dedicated endpoint | You own the endpoint and its domain | ## Before you begin ### Tensorlake requirements You need a Tensorlake project with **Self-hosted Compute** enabled. You also need permission to change the project settings and create a project-admin API key. If the Self-hosted Compute page is not available, contact Tensorlake. The same project-admin API key can prepare the Sandbox Proxy and enroll the first executor. Treat the key as a secret. Do not save it in an EC2 user-data script. ### AWS account requirements Prepare these resources in the AWS Region where you want to run sandboxes. Both modes need: * A VPC. * An S3 bucket for runtime data, built images, and durable sandbox snapshots. * An EC2 instance profile that can use the project bucket. * Capacity for at least one supported bare-metal EC2 instance: `i7i.metal-24xl` or `i7i.metal-48xl`. Compute and ingress in your network also needs: * An EKS cluster. * The [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html), version `2.7.2` or later. * An IAM role that gives the controller permission to create an AWS Network Load Balancer (NLB), target groups, and security-group rules. * A public DNS zone for the sandbox domain. * An AWS Certificate Manager (ACM) certificate for the sandbox domain. ### Sandbox base domain If you run the ingress, choose the base domain before you create certificates or manifests. Every sandbox receives a subdomain below this base domain. For example, a base domain of `sandboxes.customer.example` produces URLs such as `https://.sandboxes.customer.example`. The public endpoint must be an HTTPS origin. Do not add a port or path. Tensorlake provides the domain in the compute-only mode. ## 1. Prepare project storage Use a dedicated bucket for the project. It is the durable home for sandbox state, and executors write built sandbox images and snapshots into it. Keeping the bucket in the executor Region reduces latency and AWS data transfer costs. Block public access to the bucket. Enable encryption. If you use an AWS Key Management Service (AWS KMS) customer-managed key, give the executor instance role permission to use that key. The executor instance role needs these S3 permissions for the bucket: | Scope | Required actions | | --------------------- | ----------------------------------------------------------------------------------------------------------- | | Bucket | `s3:GetBucketLocation`, `s3:ListBucket`, `s3:ListBucketMultipartUploads` | | Objects in the bucket | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:AbortMultipartUpload`, `s3:ListMultipartUploadParts` | The enrollment preflight writes, reads, and deletes a small object under each derived storage path. This test stops the installation before a sandbox starts if the instance role cannot use the storage. Attach the role through an EC2 instance profile. Do not put static AWS access keys on the host. The Sandbox Proxy does not need this role or any other S3 permission. Open the project settings in [Tensorlake Cloud](https://cloud.tensorlake.ai). Find **Node configuration** and set the bucket URI: ```text theme={null} s3://customer-tensorlake ``` Save the node configuration before you enroll an executor. Tensorlake derives the runtime object, image-build, and snapshot paths from the bucket URI. ## 2. Prepare your network ### Subnet layout If using VPC CIDR `10.90.0.0/16`, you can set up this subnet layout, one subnet per Availability Zone: | Subnet | Ranges | Purpose | Needed by | | -------- | ------------------------------------------------- | ------------------------- | ----------------------- | | Public | `10.90.0.0/24`, `10.90.1.0/24`, `10.90.2.0/24` | NLB | Ingress in your network | | Private | `10.90.16.0/20`, `10.90.32.0/20`, `10.90.48.0/20` | Kubernetes nodes and pods | Ingress in your network | | Executor | `10.90.64.0/24`, `10.90.65.0/24`, `10.90.66.0/24` | Bare-metal EC2 instances | Both modes | If you peer with Tensorlake, use the `10.90.0.0/16` range. Tensorlake standardizes on it so that every peered deployment shares one network layout. Give each executor subnet its own route table, because you add return routes to them later. If you run the ingress, tag all three public subnets with `kubernetes.io/role/elb=1` so the AWS Load Balancer Controller can select them without relying on route-table discovery. See the [Amazon EKS NLB requirements](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html). ### Outbound access Allow outbound TCP `443` from the executor hosts, and from the EKS nodes if you run the ingress, to Tensorlake services and required AWS services. The EKS nodes must pull the public Sandbox Proxy image from `public.ecr.aws`. The executor must reach EC2 instance metadata through IMDSv2 and must reach the configured S3 bucket. Never expose executor ports to the internet in either mode. mTLS means mutual Transport Layer Security: both endpoints use certificates to verify each other. ## 3. Connect your deployment mode Complete the section for the mode you chose. Do not combine them. ### Connect compute in your network #### Send your account details Give Tensorlake these values through your support channel: | Value | Example | | ---------------------- | ------------------------------------------------- | | AWS account ID | `111111111111` | | AWS Region | `us-east-1` | | VPC ID | `vpc-0123456789abcdef0` | | VPC address range | `10.90.0.0/16` | | Executor subnet ranges | `10.90.64.0/24`, `10.90.65.0/24`, `10.90.66.0/24` | The executor subnet ranges are the destinations Tensorlake routes to. List the exact subnets, not the whole VPC range. #### Receive the onboarding packet Tensorlake creates the peering connection from its side and sends you a packet with these fields: | Field | What it is | | ----------------------- | -------------------------------------------------------------------------------------------- | | `peering_connection_id` | The connection to accept | | `requester_account_id` | The Tensorlake AWS account | | `requester_vpc_id` | The Tensorlake VPC | | `provider_node_cidrs` | One `/27` per Availability Zone, so three ranges. These are the only sources the proxy uses. | | `destination_cidrs` | The executor subnets you sent, echoed back | | `destination_ports` | TCP `5565` and TCP `8095-8097` | Verify the account, VPC, ranges, and ports before you accept anything. #### Accept the peering connection ```bash theme={null} aws ec2 accept-vpc-peering-connection \ --region "$AWS_REGION" \ --vpc-peering-connection-id "$PEERING_CONNECTION_ID" ``` Confirm the status: ```bash theme={null} aws ec2 describe-vpc-peering-connections \ --region "$AWS_REGION" \ --vpc-peering-connection-ids "$PEERING_CONNECTION_ID" \ --query 'VpcPeeringConnections[0].Status.Code' \ --output text ``` The status must be `active`. #### Route return traffic Add a route in all three executor subnet route tables for each `provider_node_cidrs` value, with the peering connection as the target. Traffic that arrives from the proxy has no return path until you do. #### Allow the proxy into the executor security group Allow each `provider_node_cidrs` value into every executor security group on TCP `5565` and TCP `8095-8097`. Allow the `/27` node ranges only. Do not allow a pod range, and do not allow the whole Tensorlake VPC range. If restrictive network ACLs protect the executor subnets, allow both request and return traffic there as well. ### Connect compute and ingress in your network #### Create the load balancer security group The public NLB terminates TLS and sends traffic to the Sandbox Proxy pods. | Direction | Port | Source or target | Purpose | | --------- | -----: | ------------------------------------ | ----------------------------- | | Inbound | `443` | Internet or approved client networks | HTTPS, WebSocket, and CLI SSH | | Inbound | `22` | Approved SSH client networks | Optional native SSH | | Outbound | `9080` | Sandbox Proxy pod security group | Proxy HTTP traffic | | Outbound | `2222` | Sandbox Proxy pod security group | Proxy SSH traffic | The generated Service asks the AWS Load Balancer Controller to manage the matching backend security-group rules. The controller IAM role must include permission to add and remove those rules. #### Allow the proxy into the executor security group Allow traffic from the Sandbox Proxy pod network or security group to the executor security group: | Port | Purpose | | -----: | -------------------------------------------- | | `8096` | Native SSH traffic to a sandbox | | `8097` | Dataplane API traffic over mutual TLS (mTLS) | | `5565` | Raw sandbox TCP traffic over mTLS | #### Create the public certificate Request an ACM certificate in the same AWS account and Region as the NLB. Add both names to the certificate: ```text theme={null} sandboxes.customer.example *.sandboxes.customer.example ``` Create the DNS validation records. Wait until ACM shows the certificate status as `ISSUED`. #### Prepare the Sandbox Proxy The preparation script creates the customer-specific mTLS certificate, two SSH key pairs, and a local Kustomize overlay. Kustomize is a Kubernetes manifest customization tool. The script creates files only in the output directory. It does not change your cluster or AWS account. Download and run the script: ```bash theme={null} curl -fsSL \ https://cloud.tensorlake.ai/byoc/sandbox-proxy/prepare.sh \ --output prepare.sh chmod +x prepare.sh ./prepare.sh \ --api-key \ --endpoint https://sandboxes.customer.example \ --namespace tensorlake-sandbox-proxy \ --output-dir ./tensorlake-sandbox-proxy \ --enable-aws-lb \ --aws-lb-certificate-arn '' \ --aws-lb-security-group-id '' ``` Keep the output directory secure. It contains private mTLS and SSH keys. Do not commit it to a source repository. #### Set the SSH proxy public key In the project settings in [Tensorlake Cloud](https://cloud.tensorlake.ai), set **SSH proxy public key** to the content of this generated file: ```text theme={null} ./tensorlake-sandbox-proxy/ssh/outbound.key.pub ``` The Sandbox Proxy uses the matching private key when it connects to a sandbox SSH server. Leave this setting empty only if you want to disable SSH for the project. Tensorlake owns this key in the compute-only mode, so it is set only when you run the ingress yourself. #### Deploy the Sandbox Proxy Confirm the active Kubernetes context: ```bash theme={null} kubectl config current-context ``` Review the generated manifests and run a server-side dry run: ```bash theme={null} kubectl kustomize ./tensorlake-sandbox-proxy kubectl apply \ --server-side \ --dry-run=server \ -k ./tensorlake-sandbox-proxy ``` Add a Deployment patch if the cluster requires a node selector or toleration. Then apply the resources: ```bash theme={null} kubectl apply -k ./tensorlake-sandbox-proxy kubectl \ --namespace tensorlake-sandbox-proxy \ rollout status deployment/sandbox-proxy ``` Wait for the controller to publish the NLB hostname: ```bash theme={null} kubectl \ --namespace tensorlake-sandbox-proxy \ get service sandbox-proxy-public ``` Confirm that both NLB target groups are healthy. The listener on port `443` uses pod port `9080`. The listener on port `22` uses pod port `2222`. #### Create DNS records and activate the endpoint Create two public DNS alias or CNAME records. Point both records to the NLB hostname: ```text theme={null} sandboxes.customer.example *.sandboxes.customer.example ``` You can use Amazon Route 53 or another public DNS provider. Open the project settings in Tensorlake Cloud. Set **Sandbox ingress endpoint** to this value: ```text theme={null} https://sandboxes.customer.example ``` Save the setting. Verify the public health route: ```bash theme={null} curl -fsS https://sandboxes.customer.example/health ``` Verify that the certificate also covers sandbox subdomains: ```bash theme={null} openssl s_client \ -connect sandboxes.customer.example:443 \ -servername test.sandboxes.customer.example \ /dev/null \ | openssl x509 -noout -checkhost test.sandboxes.customer.example ``` The command prints `Hostname test.sandboxes.customer.example does match certificate`. ## 4. Launch the first executor Launch a bare-metal EC2 instance with Ubuntu `24.04`. Attach the instance profile from the storage step. The host must expose `/dev/kvm` because Tensorlake uses hardware virtualization to isolate sandboxes. The instance must also have local NVMe instance storage. Enrollment formats those disks and mounts them at `/mnt/data`, which holds container images, sandbox images, and the local snapshot cache. Amazon EBS volumes are never used for this, so an instance type without local NVMe cannot run sandboxes. These instance types meet both requirements: ```text theme={null} i7i.metal-24xl i7i.metal-48xl ``` Place the instance in an executor subnet that can receive Sandbox Proxy traffic. Enable IMDSv2. Give the root volume 50 GB so it holds the operating system and the downloaded components. ## 5. Enroll the executor In Tensorlake Cloud, open **Self-hosted Compute**, select **Add node**, and copy the bootstrap command. It has this form: ```bash theme={null} curl -fsSL https://cloud.tensorlake.ai/byoc/bootstrap.sh \ | sudo bash -s -- --api-key ``` Run the command on the EC2 instance. The bootstrap process checks the host and S3 access. It then creates the executor identity, installs the provisioner, and starts the executor services. The project-admin API key is used only during enrollment. The installed provisioner uses its own mTLS identity after enrollment. It continues to check Tensorlake for approved configuration and software updates. Return to **Self-hosted Compute**. Confirm that the new node appears and becomes ready before you create a sandbox. ## 6. Verify the complete path Install the Tensorlake CLI if the `tl` command is not available: ```bash theme={null} curl -fsSL https://tensorlake.ai/install | sh ``` Sign in and select the BYOC project: ```bash theme={null} tl login tl init ``` Create a sandbox. Use its ID in the remaining commands: ```bash theme={null} tl sbx create tl sbx exec sh -lc 'printf BYOC_OK' tl sbx ssh ``` Every sandbox gets a subdomain of the project endpoint. If you run the ingress, that is your own base domain: ```text theme={null} https://.sandboxes.customer.example ``` Test native SSH if port `22` is enabled: ```bash theme={null} tl sbx ssh keys add ~/.ssh/id_ed25519.pub ssh @.sandboxes.customer.example ``` Create a snapshot and confirm that the executor writes data into the bucket: ```bash theme={null} tl sbx snapshot ``` ## Add more executors Each executor needs its own EC2 instance and enrollment identity. Launch another supported host with the same network and instance-profile configuration. Then run a new command from **Self-hosted Compute → Add node**. The project storage and Sandbox Proxy settings apply to all executors in the project. You do not need another proxy deployment or peering connection for each host. ## Updates and key rotation The provisioner runs as a service on every enrolled executor. It checks Tensorlake for new approved profiles and applies complete updates. A failed update keeps the last valid revision active. If you run the ingress, rotate the Sandbox Proxy mTLS and SSH keys by running `prepare.sh` again with a new output directory. Review and apply the new resources. The certificate fingerprint starts a rolling pod update. Then replace **SSH proxy public key** with the new `outbound.key.pub` value. Keep the previous output directory in secure storage until the rollout and end-to-end test pass. ## Troubleshooting Both modes: | Symptom | Check | | ------------------------------------------------ | ----------------------------------------------------------------------------------- | | The executor enrollment stops during host checks | Confirm Ubuntu `24.04`, root access, IMDSv2, and `/dev/kvm`. | | The storage preflight fails | Check the EC2 instance profile, S3 actions, bucket policy, and AWS KMS permissions. | | A sandbox remains pending | Confirm that the executor is ready and belongs to the same Tensorlake project. | Compute in your network: | Symptom | Check | | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | The peering connection stays `pending-acceptance` | Accept it in the Region and account named in the packet. | | Sandboxes never become ready | Confirm the return routes in every executor subnet route table, not only the first. | | Traffic arrives but nothing answers | Confirm the executor security group allows every `provider_node_cidrs` value on TCP `5565` and TCP `8095-8097`. | | Some Availability Zones work and others do not | The proxy uses one node range per zone. Confirm all of them appear in your routes and security groups. | | The executor registers with a public address | Confirm the instance takes its address from instance metadata and sits in an executor subnet you sent. | Compute and ingress in your network: | Symptom | Check | | -------------------------------- | -------------------------------------------------------------------------------------------- | | The NLB is not created | Check the AWS Load Balancer Controller, its IAM role, and public subnet discovery. | | An NLB target is unhealthy | Check pod readiness and traffic from the NLB security group to pod port `9080`. | | The proxy cannot reach a sandbox | Check private routes and executor security-group rules for ports `8096`, `8097`, and `5565`. | | HTTPS works but native SSH fails | Check the NLB listener on port `22`, its target group, and both SSH security-group paths. |