Automated Game Servers with Terraform and Docker #

Automating my game server setup makes it a lot faster, cheaper, and easier to stand up a game server that will only live for maybe a few hours before being destroyed. Before, I was paying around 10-20 dollars a month to setup servers that were only being used for a few hours every week. But with Terraform and Docker, I get servers that can be created in only a few minutes and then destroyed after the session is over.

The high level overview is to use Terraform to provision our infrastructure. It will handle setting up the instances, updating our DNS records, and setting up any security groups or launch templates. When we setup the instance, we will configure it with a user data script that will update the system, install docker, and setup any state that we can download from S3.

The largest issue with this kind of provisioning is that the docker images I currently use download the entire game server every single time. This can take upwards of twenty minutes for game servers like CS:GO or Gmod that need to download the entire game with SteamCMD. We can probably cache this somewhere locally to speed things up, but ingress is usually free so I didn’t implement it.

Terraform Requirements #

To start using Terraform, you need to setup some access keys. I’m using AWS for this, but the same applies to all public clouds. In AWS, create an IAM user and either grant it the following policies, or create your own custom policy for more fine grained control. In a real environment you would want to lock things down more and not just grant the terraform user full access to everything.

- AmazonEC2FullAccess
- IAMFullAccess
- AmazonS3 FullAccess
- AmazonRoute53FullAccess

Then download the access key and secrets and setup aws-cli on the computer that you want to orchestrate things from. Just install aws-cli and then run aws configure.

Writing a Terraform deployment script #

Create a folder to hold your terraform scripts and create a main.tf as follows:

variable "name" {
}

variable "type" {
}

variable "spot_price" {
}

variable "aws_access_key_id" {
}

variable "aws_secret_access_key" {
}

variable "aws_default_region" {
	default = "us-west-2"
}

variable "terraform_pub_key_file" {
	default = "~/.ssh/id_ed25519.pub"
}

provider "aws" {
    version = "~> 2.0"
    region = var.aws_default_region
    access_key = var.aws_access_key_id
    secret_key = var.aws_secret_access_key
}

resource "aws_iam_instance_profile" "profile" {
  name = "terraform_profile"
  role = aws_iam_role.role.name
}

resource "aws_iam_role" "role" {
  name = "terraform_role"
  path = "/"

  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
               "Service": "ec2.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": ""
        }
    ]
}
EOF
}

resource "aws_iam_role_policy" "policy" {
  name = "terraform_policy"
  role = aws_iam_role.role.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

resource "aws_security_group" "sg" {
    name = format("terraform_%s", var.name)
    ingress {
        from_port = 22
        to_port   = 22
        protocol  = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }

    ingress {
        from_port = 25565
        to_port   = 25565
        protocol  = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }

    # Allow all outgoing
    egress {
        from_port = 0
        to_port   = 0
        protocol  = "-1"
        cidr_blocks = ["0.0.0.0/0"]
        ipv6_cidr_blocks = ["::/0"]
    }
}

resource "aws_spot_instance_request" "worker" {
    ami = "ami-003634241a8fcdec0" // Ubuntu 18.04 LTS
    instance_type = var.type

    security_groups = [ aws_security_group.sg.name ]
    key_name = "aws"
    iam_instance_profile = aws_iam_instance_profile.profile.name
    user_data = file("setup.sh")

    spot_price = var.spot_price
    spot_type = "one-time"
    wait_for_fulfillment = "true"
}

resource "aws_route53_record" "record" {
    zone_id = "put your route53 zone ID here"
    name = format("%s.example.com", var.name) // Replace with your domain name
    type = "A"
    ttl = "300"
    records = [ aws_spot_instance_request.worker.public_ip ]
}

Notice that there are a bunch of variables that define the secrets and stuff we don’t want to commit. We tell terraform these variables exist with a variable declaration block. For more information, see https://www.terraform.io/docs/configuration/variables.html

Provisioning #

The most important part of this process is the setup.sh script that is passed as userdata to the instance creation request. This script is responsible for provisioning everything after the instance is created. By passing it in as userdata and not through terraform directly, we don’t need any kind of remote access to the provisioned server and the userdata scripts also support autoscaling if you need that in the future. This setup script can also be used to install a real provisioning tool like ansible, salt, or chef once the provisioning steps get more complicated. For now, an example setup.sh script for a minecraft server that backs up its data on S3 is as follows:

#!/bin/bash

# Terminate on any error, terminate pipes preemptively on any error, output commands to stdout
set -e
set -o pipefail
set -x

# Update system
echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4
DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null
DEBIAN_FRONTEND=noninteractive apt-get -y upgrade >/dev/null

# Install docker and other pre-reqs
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq >/dev/null \
	apt-transport-https \
	ca-certificates \
	curl \
	gnupg-agent \
	software-properties-common \
    awscli \
    unzip
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
	"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
	$(lsb_release -cs) \
	stable"
DEBIAN_FRONTEND=noninteractive apt-get update -qq >/dev/null
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq >/dev/null docker-ce docker-ce-cli containerd.io

# Install docker-compose
curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# Download world and allow it to fail if the world does not exist in the bucket
mkdir -p /root/data/world || true
/usr/bin/aws s3 cp s3://putyourbucketnamehere/world.tar.gz /root/world.tar.gz || true
tar -xzf /root/world.tar.gz -C /root/data/ || true
rm /root/world.tar.gz || true

# Install mods
mkdir -p /root/data/mods
/usr/bin/aws s3 cp s3://putyourbucketnamehere/mods-server.zip /root/data/mods/mods.zip
unzip -oqq /root/data/mods/mods.zip -d /root/data/mods/
rm /root/data/mods/mods.zip

# Setup save and quit script
cat > /root/cleanup.sh << EOF
#!/bin/bash
docker exec minecraft.service rcon-cli stop
sleep 5
tar -czf /root/world.tar.gz -C /root/data/ world
/usr/bin/aws s3 mv s3://putyourbucketnamehere/world.tar.gz s3://putyourbucketnamehere/world.tar.gz.bk
/usr/bin/aws s3 cp /root/world.tar.gz s3://putyourbucketnamehere/world.tar.gz
EOF
chmod +x /root/cleanup.sh

cat > /etc/systemd/system/minecraft.service <<EOF
[Unit]
Description=Docker Server Container
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm -f %n
ExecStart=/usr/bin/docker run --name %n \
              --restart=on-failure:10 \
			  --network host \
              --log-driver=journald \
			  --mount type=bind,source=/root/data,target=/data \
			  -e VERSION=1.14.4 \
			  -e TYPE=FORGE \
			  -e EULA=TRUE \
			  -e SERVER_NAME="Minecraft Server" \
			  -e DIFFICULTY=hard \
			  -e OPS="example" \
			  -e MAX_PLAYERS=5 \
			  -e SPAWN_PROTECTION=0 \
			  -e MOTD=kys \
			  -e LEVEL_TYPE=flat \
			  -e RCON_PASSWORD="Example Password" \
			  -e WHITELIST="example" \
			  -e ALLOW_FLIGHT=TRUE \
			  -e MEMORY=13G \
			  -e USE_AIKAR_FLAGS=true \
			  itzg/minecraft-server
ExecStop=-/usr/bin/docker stop %n

[Install]
WantedBy=default.target
EOF

# Setup hook for spot instance termination notice
cat > /root/termination-check.sh << EOF
#!/bin/bash
while true; do
    echo "Checking for spot instance termination notice..."
    if curl -s http://169.254.169.254/latest/meta-data/spot/instance-action | grep -q .*T.*Z; then
        /root/cleanup.sh
        break
    fi

    sleep 5
done
EOF

systemctl enable minecraft.service
systemctl start minecraft.service

# Start the termination notice hook
bash /root/termination-check.sh

Deploying #

Then just run terraform plan to see what it will do and then run terraform apply to setup and create everything. To destroy everything completely, run terraform destroy, but be careful and make sure that all your persistent data is backed up before destroying. After a few minutes, everything should be up and running for just a few cents every hour. For example, I ran a heaily modded FTB server on a m5.large instance for just 0.0346 cents per hour. As long as you don’t need to run the server 24/7 for weeks, then the spot instance stuff probably won’t affect you and you save a lot. If I ran the equivalent dedicated cpu server on Linode for one month, it would be about 0.09 cents per hour.