When working with Terraform, HashiCorp recommends keeping your workspaces small and focused on the resources that make up a single component of a larger infrastructure stack. Doing so has many benefits, but this best practice can introduce dependencies between workspaces, which in turn introduces a new challenge: how do you ensure that these interdependent component workspaces are automatically created (or destroyed) in the right order?

I wrote this blog post to present a pattern to reduce operational overhead from managing multi-workspace deployments. When combined with ephemeral workspaces, a feature coming soon to Terraform Cloud, this pattern can also help reduce costs by allowing you to destroy an entire stack of workspaces in a logical order. I’m a HashiCorp Solutions Engineer who uses this pattern frequently, and it’s used by Terraform creator and HashiCorp Co-Founder Mitchell Hashimoto, and many others.

Notes: This method is not an official HashiCorp-recommended best practice and is not intended to be the solution to all use cases. Rather, it’s just one example to explore and build upon. This blog post also includes a simpler solution to the challenge above, which should work for a large number of use cases. While this blog post was written with Terraform Cloud in mind, the same concepts and configuration will also work in Terraform Enterprise as well, depending on your version. Finally, the suggestions presented here all assume a basic understanding of Terraform Cloud, and you can try out the code examples yourself in this GitHub repository.

Workspaces, run triggers, and the Terraform Cloud/Enterprise provider

A workspace in Terraform Cloud contains everything Terraform needs to manage a given collection of infrastructure, including variables, state, configuration, and credentials. For example, you may have one workspace that defines your virtual network and another for compute. These workspaces depend on each other; you cannot create your compute until you have a virtual network. The outputs from one workspace become the inputs to another.

Terraform Cloud has a feature called run triggers. This allows you to kick off an apply on your compute workspace after your virtual network has been created. You can also define a one-to-many relationship here, where the successful apply on one upstream workspace triggers multiple downstream workspaces.

This alone solves some dependency challenges and is great for simple workflows. If it works for your use case, you should use it.

However, it doesn’t address all situations, which is why Mitchell Hashimoto created the multispace provider to handle the kind of cascading creation/destruction workflows that can’t be done with simple run triggers. His example use case involves creating a Kubernetes stack: first you create the underlying virtual machines, then the core kubernetes services, DNS, and ingress. Each of these is its own separate workspace.

This initial implementation has since been refined and incorporated into the official Terraform Cloud/Enterprise provider (also called the “TFE provider”) in the form of the tfe_workspace_run resource.

For clarity, this post uses the following terminology when referring to the roles a workspace could have in multi-workspace deployments (an individual workspace may have one or more of these roles):

• Upstream workspace: This is a workspace that must run first, and is depended upon in some way by a downstream workspace
• Downstream workspace: This is a workspace that runs second, because it has some dependency on the upstream workspace
• Workspace runner: This is a workspace responsible for triggering runs on other workspaces with the tfe_workspace_run resource.
• Workspace creator: This is a workspace responsible for creating other workspaces with the tfe_workspace resource. This is usually also a workspace runner in my use cases, but it may not be a requirement for yours.

Applies

Here’s an example of how run triggers and tfe_workspace_run differ: run triggers will always kick off a plan on the downstream workspace once the upstream workspace has completed a successful apply. Sometimes this results in plans on the downstream workspace that are unnecessary (in the case of a do-nothing plan) or that fail (when a downstream workspace has a dependency on multiple upstream workspaces but some upstream workspaces haven’t yet completed their apply phases).

With tfe_workspace_run you can specify when to apply and under what circumstance. For example, with depends_on, a workspace runner could wait until several upstream workspaces have applied before kicking off the downstream workspace. If that is the only benefit relevant to you, chances are that run triggers are probably good enough for your use case; you’re probably fine with a do-nothing or failed plan every now and then.

You can use the tfe_workspace_run resource in two operational modes:

Fire-and-forget: The resource simply queues a run on the downstream workspace and considers it good enough if the run was successfully queued. This mode is very similar to how run triggers work.

Wait: The resource queues a run on the downstream workspace and waits for it to successfully apply. After a successful plan, the resource can wait for a human approval on the apply or initiate the apply itself. Optionally, the resource can retry if the apply fails.

Destroys

Run triggers do one thing: they trigger apply runs on downstream workspaces, and they do it only after the upstream has completed successfully. They do not handle destruction use cases. For example, you should destroy your compute before destroying your virtual network, and run triggers do not give you a means to model that side of the dependency.

This is where the real power of tfe_workspace_run comes in. The resource allows you to kick off a destroy on a downstream workspace and, if you’re using depends_on, you can ensure that nothing in the upstream workspace is destroyed until the downstream workspace has successfully finished its destroy.

Apply-only and destroy-only

While you can configure both the apply and destroy behavior for the downstream workspace, you don’t need to use both. There are cases where you only want to apply a downstream workspace. There are also times where you only want to destroy a downstream workspace, but you will trigger an apply yourself.

tfe_workspace_run in action

The tfe_workspace_run resource documentation on the Terraform Registry includes a few example code snippets to use as a starting point. At its most basic, the resource looks like this:

resource "tfe_workspace_run" "ws_run_parent" {
  workspace_id = "ws-fSX576JZGENVaeMi"
 
  apply {
    # tfe_workspace_run is responsible for approving the apply
    # part of the run
    # this is the only required argument in the apply{} and
    # destroy{} blocks
    manual_confirm    = false
 
    # if the run fails, try again, up to 3 times, waiting between
    # 1 and 30 seconds
    # this is the default behaviour, presented here for clarity
    wait_for_run      = true
    retry_attempts    = 3
    retry_backoff_min = 1
    retry_backoff_max = 30
  }
 
  destroy {
    manual_confirm    = false
  }
}

This example shows what’s meant by a workspace runner. For the specified workspace, our tfe_workspace_run resource will trigger an apply, wait for that to complete, then consider the tfe_workspace_run successfully created. On destroy, the tfe_workspace_run will trigger a destroy, wait for that to complete, then consider the tfe_workspace_run successfully destroyed.

Because we are using the TFE provider, the workspace runner requires a TFE_TOKEN with sufficient permissions to kick off plan/apply/destroy runs on child workspaces. (You may wish to use the Terraform Cloud secrets engine in HashiCorp Vault to generate these, but that is out-of-scope for this blog post.)

Beyond the basic examples, this post will present a few patterns with example configuration for how to use this resource.

Apply (fire and forget) and destroy (wait)

The tfe_workspace_run resource is most useful when creating new workspaces. This example uses the TFE provider to create a workspace, set up all the necessary permissions, and configure the dynamic credentials. All of that must be done before the workspace can be applied.

As a reminder, the term “workspace creator” refers to any workspace responsible for creating other workspaces and related resources. In most cases when using a workspace creator, it will also be a workspace runner for the workspaces it creates (i.e. it is responsible for triggering apply and/or destroy runs on those workspaces).

resource "tfe_workspace_run" "downstream" {
  workspace_id = tfe_workspace.downstream.id
 
  # depends_on = creds and other workspace dependencies go here
 
  apply {
    # Fire and Forget
    wait_for_run = false
    # auto-apply
    manual_confirm = false
  }
 
  destroy {
    # Wait for destroy before doing anything else
    wait_for_run = true
    # auto-apply
    manual_confirm = false
  }
}

From the perspective of the workspace runner, it doesn’t need to care if the downstream workspace was successfully applied, just that an apply was attempted. This functionality alone is achievable with run triggers (and in this fire-and-forget mode, the behavior is very similar), but as this example is already using tfe_workspace_run to handle the destroy, it makes sense to use it for the apply as well.

By including the destroy{} block in combination with depends_on, you can ensure that the workspace and supporting resources remain untouched until the downstream workspace has successfully destroyed all the resources it manages.

If you do not include a destroy{} block, then attempting to delete the downstream workspace will result in an error like this:

Error: error deleting workspace ws-BxxKPnyBVpxwVQB1: This workspace has 4 resources under management and must be force deleted by setting force_delete = true

If you do not include the depends_on, then dependencies such as variables and credentials that the downstream workspace needs will end up getting deleted too early.

The upcoming ephemeral workspaces will ensure the entire stack of workspaces is safely destroyed in the correct order once the ephemeral workspace hits its time-to-live (TTL).

Destroy-only workflows

This is a pattern I use extensively in my workspace creator. As a reminder, this is a “workspace runner” (i.e. a workspace which triggers runs on other workspaces). In my case, when creating a new workspace, I don’t necessarily want it to try to apply immediately, so my configuration looks like this:

resource "tfe_workspace_run" "downstream" {
  workspace_id = tfe_workspace.downstream.id
 
  # depends_on = creds and other workspace dependencies go here
 
  destroy {
    wait_for_run = true
    manual_confirm = false
  }
}

The only difference between this and the previous example is the absence of the apply{} block. This means that when doing an apply on the workspace runner, Terraform will create a placeholder resource referencing a non-existent apply on the downstream workspace. This may seem pointless as nothing has actually been done yet, but the presence of this resource in Terraform state signals to Terraform that, when it comes time to destroy the workspace runner, it should first kick off a destroy on the downstream workspace.

Similarly, you can also have an apply-only workflow by including an apply{} block but no destroy{}. For most use cases, an apply{} block with no destroy{} block is practically identical to just using run triggers. If it is, then just use run triggers. But for some niche use-cases, you may want to conditionally apply, which run triggers does not allow you to do.

What if there are more than two workspaces?

This is the main reason for the concept of a workspace runner separate from the idea of an upstream workspace. The previous examples have a single upstream workspace for every downstream workspace. In cases like that, introducing an additional workspace runner just adds unnecessary complexity; the upstream can handle the runner functionality.

Workspace runners become useful in cases where there are more than two workspaces. While upstream workspaces can handle the runner role functionally, if you have apply or destroy runs configured to wait for completion, then you’ll have more workspaces in your stack, which results in more concurrent runs. If you have too many, the queue will fill up, and you’ll end up in a deadlock, where downstream workspaces are queued but can never begin, and upstream workspaces are waiting on those downstream workspace runs.

By introducing a separate workspace runner, you can ensure you need to consume only two concurrency slots: one for the runner, and one for whichever other workspace it is currently running.

Some examples, first a simple one, then a complex one:

resource "tfe_workspace_run" "A" {
  workspace_id = data.tfe_workspace.ws["A"].id
 
  apply {
    wait_for_run   = true
    manual_confirm = false
  }
 
  destroy {
    wait_for_run   = true
    manual_confirm = false
  }
}
 
resource "tfe_workspace_run" "B" {
  workspace_id = data.tfe_workspace.ws["B"].id
 
  depends_on = [tfe_workspace_run.A]
 
  apply {
    wait_for_run   = true
    manual_confirm = false
  }
 
  destroy {
    wait_for_run   = true
    manual_confirm = false
  }
}
 
resource "tfe_workspace_run" "C" {
  workspace_id = data.tfe_workspace.ws["C"].id
 
  depends_on = [tfe_workspace_run.B]
 
  apply {
    wait_for_run   = true
    manual_confirm = false
  }
 
  destroy {
    wait_for_run   = true
    manual_confirm = false
  }
}

• In this example, to apply, the workspace runner queues a run on A, waits for it to complete, queues a run on B, waits for it to complete, then queues a run on C.
• If you care only about the apply phase, this is a perfect use case for run triggers.
• To destroy, the workspace runner queues a run on C first, then B, then A.

Of course, there’s no reason you’re limited to each workspace having one upstream or one downstream. You may have a complex web of dependencies between your workspaces. Here’s as example (with the code in the repo linked at the end of the post):

• Whether or not there’s an actual real-world use case for this precise graph of workspace dependencies, it serves as a good example of the complexity you can end up with.
• In this example, the workspace runner queues runs on U1, U2, and U3.
• If there are spare concurrency slots, all three will run concurrently. If not, at least one will wait in the queue.
• Once all three have finished, then the workspace runner queues runs on D1, D2, and D3.
• Again, if you just care about the apply phase, run triggers would work, but you would get a lot of failed plans because U1, U2, and U3 finish at different times.
• To destroy, the workspace runner queues a run on D1, D2, and D3 first, then U1, U2, and U3.

Summary

The resource required to power this workflow is now an official, supported part of the TFE provider. As more people make use of tfe_workspace_run and give feedback, HashiCorp can continue to improve the resource.

If you aren’t already using Terraform Cloud, you’ll want to experiment with that first. Start for free by signing up for an account.

Please note that to avoid hitting concurrency limits on the Free tier of Terraform Cloud, you can:

• Use only run triggers or tfe_workspace_run in fire-and-forget mode, or
• Ensure the workspace runner is running in local mode if you plan to use tfe_workspace_run in wait mode, or
• Upgrade to a paid subscription to increase your concurrency limits.

If you are an existing Terraform Cloud user, then experiment with these patterns. Regardless of whether you’re using run triggers or tfe_workspace_run, automating the dependencies between workspaces will save you time and effort.

As mentioned at the beginning of the post, you can find these code examples in this GitHub repository.

In the beginning…

So this whole story starts with me looking to manage my DNS records better.

I wrote a whole blog post about it if you’re interested: Moving my DNS records to Route 53 with Terraform

I was modifying them by hand in our domain registrar’s own DNS settings page… and it was kinda a pain.

At work we use AWS Route 53 for some DNS records, so I was familiar with how that worked, and it seemed like a good idea to move to that too. Not because managing DNS records in Route 53 by hand is much easier, but it opened up the possibility for automating it.

Terraform!

Which is where Terraform comes in!

Before we can create some DNS records, we need a hosted zone

It’s prety simple to create one. Just needs a name:

resource "aws_route53_zone" "lmhd-me" {
  name = "lmhd.me"
}

I will want to delegate the test subdomain to a different zone too:

resource "aws_route53_record" "lmhd-me_NS_test-lmhd-me" {
  zone_id = aws_route53_zone.lmhd-me.zone_id
  name    = "test.lmhd.me"
  type    = "NS"
  ttl     = "300"

  records = aws_route53_zone.test-lmhd-me.name_servers
}

I don’t need to do that, but I like that separation, and it gives me the flexibilty to move that to an entirely different AWS account (or somewhere else) in future.

The first problem I encounted almost immediately was that you can’t do CNAMEs on apex domains, and I wanted to do something for lmhd.me.

Route 53 doesn’t have a concept of an “ANAME”. So I would have to roll my own.

Thankfully, Terraform has a DNS Provider so I could query for the IP Address of my Netlify site, and then use that when defining my A Record for lmhd.me:

# Lookup CNAME for Netlify.
data "dns_cname_record_set" "lmhd-dot-me-netlify-com" {
  host = "lmhd-dot-me.netlify.com"
}

# Get IPs for that CNAME
data "dns_a_record_set" "netlify-com" {
  host = data.dns_cname_record_set.lmhd-dot-me-netlify-com.cname
}

# Set up the A record for lmhd.me
resource "aws_route53_record" "lmhd-me-A-record" {
  zone_id = aws_route53_zone.lmhd-me.zone_id
  name    = "lmhd.me"
  type    = "A"
  ttl     = "300"

  records = ["${data.dns_a_record_set.netlify-com.addrs}"]
}

The rest of the records are fairly simple. Most are just CNAMEs, and have similar format to the above.

I set this up to run in Terraform Cloud, and away I went, adding new DNS records as and when I needed them. For example:

resource "aws_route53_record" "widgets-lmhd-me-CNAME-record" {
  zone_id = aws_route53_zone.lmhd-me.zone_id
  name    = "widgets.lmhd.me"
  type    = "CNAME"
  ttl     = "300"

  records = ["lmhd-widgets.netlify.app"]
}

Okay, this is getting tedious

I’m lazy efficient, so I don’t want to have to write out all of that every time I want to add something.

Can we make it easier on ourselves?

What if we make TF Variables out of this?

Took a bit of fiddling to get something which works, but here’s what I came up with as my initial version:

#
# DNS Records
#

variable "lmhd_records" {
  type = map(object({
    type    = string
    ttl     = string
    records = list(string)
  }))
  default = {
    "widgets.lmhd.me" = {
      type    = "CNAME"
      ttl     = "300"
      records = ["lmhd-widgets.netlify.app"]
    }
  }
}

#
# DNS Records
#

resource "aws_route53_record" "lmhd_record" {
  for_each = var.lmhd_records
  zone_id  = aws_route53_zone.lmhd-me.zone_id
  name     = each.key
  type     = each.value.type
  ttl      = each.value.ttl
  records  = each.value.records
}

This makes use of Terraform’s for_each Meta-Argument, so rather than defining a Terraform resource for every DNS record, I define a single resource and update the variable instead.

And our plan looks like this:

  # aws_route53_record.lmhd_record["widgets.lmhd.me"] will be created
  + resource "aws_route53_record" "lmhd_record" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "widgets.lmhd.me"
      + records         = [
          + "lmhd-widgets.netlify.app",
        ]
      + ttl             = 300
      + type            = "CNAME"
      + zone_id         = "REDACTED"
    }

That’s… okay. Maybe slightly less typing… but now it’s messy. Why would I want to do this?

And if I stopped there, I would agree, it’s not really worth it.

But it is a stepping stone in the right direction: getting automatically generated TF code!

Auto-generated TF code

Can I use external files to auto-generate my Terraform code?

And… yes. Terraform has a bunch of functions, including fileset which lets me query the content of a directory.

First we need to pull in some files…

locals {
  lmhd_records_files = fileset(path.module, "lmhd.me/*.json")
}

output "lmhd_records" {
  value = local.lmhd_records_files
}

This gives us a list of all JSON files in the lmhd.me/ directory:

Changes to Outputs:
  + lmhd_records = [
      + "lmhd.me/widgets.lmhd.me.json",
    ]

So that’s a start.

That one JSON file of mine looks like this:

{
  "type": "CNAME",
  "ttl": "300",
  "records": [
    "lmhd-widgets.netlify.app"
  ]
}

And we can pull that in with Terraform making use of a few other functions:

resource "aws_route53_record" "lmhd_record" {
  for_each = local.lmhd_records_files
  zone_id  = aws_route53_zone.lmhd-me.zone_id

  # Use the filename as the name of the DNS record
  # e.g. lmhd.me/widgets.lmhd.me.json --> widgets.lmhd.me
  name     = trimsuffix(basename(each.key), ".json")

  # Get the rest of the values from keys in the JSON file
  type     = jsondecode(file(each.key))["type"]
  ttl      = jsondecode(file(each.key))["ttl"]
  records  = jsondecode(file(each.key))["records"]
}

Our plan looks like this:

  # aws_route53_record.lmhd_record["lmhd.me/widgets.lmhd.me.json"] will be created
  + resource "aws_route53_record" "lmhd_record" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "widgets.lmhd.me"
      + records         = [
          + "lmhd-widgets.netlify.app",
        ]
      + ttl             = 300
      + type            = "CNAME"
      + zone_id         = "REDACTED"
    }

Looking good 🙂

But… I like YAML… 👉👈

Okay, fine…

Our widgets.lmhd.me.yaml file:

type: CNAME
ttl: 300
records:
- lmhd-widgets.netlify.app

locals {
  lmhd_records_files = fileset(path.module, "lmhd.me/*.yaml")
}

resource "aws_route53_record" "lmhd_record" {
  for_each = local.lmhd_records_files
  zone_id  = aws_route53_zone.lmhd-me.zone_id

  # Use the filename as the name of the DNS record
  # e.g. lmhd.me/widgets.lmhd.me.json --> widgets.lmhd.me
  name     = trimsuffix(basename(each.key), ".yaml")

  # Get the rest of the values from keys in the YAML file
  type     = yamldecode(file(each.key))["type"]
  ttl      = yamldecode(file(each.key))["ttl"]
  records  = yamldecode(file(each.key))["records"]
}

Happy?

Some validation

That works fine as an initial Proof of Concept… but what if I’ve not specified some of those keys in my file?

In my case, most of these values are going to be the same.

I’m primarily defining CNAMEs, and I rarely change the TTL.

So I kinda don’t really want to have to specify them in my files every time.

Can we do something about that?

And… yes we can.

Let’s have a YAML file like this:

records:
- lmhd-widgets.netlify.app

And some Terraform code like this:

variable "default_ttl" {
  type    = string
  default = "300"
}

resource "aws_route53_record" "lmhd_record" {
  for_each = local.lmhd_records_files
  zone_id  = aws_route53_zone.lmhd-me.zone_id

  # Use the "name" key if specified
  # otherwise fallback to filename, minus .yaml
  name = lookup(
    yamldecode(file(each.key)),
    "name",
    trimsuffix(basename(each.key), ".yaml")
  )

  # Default to CNAME type unless told otherwise
  type = lookup(
    yamldecode(file(each.key)),
    "type",
    "CNAME"
  )

  # Use Default TTL unless told otherwise
  ttl = lookup(
    yamldecode(file(each.key)),
    "ttl",
    var.default_ttl
  )

  # We must have some records, otherwise this will fail
  records = yamldecode(file(each.key))["records"]
}

Now, if I do not specify some of those values, it will use the defaults.

I can also override the name of the DNS record if I so chose.

That last bit, if we do not specify any records in our YAML file, Terraform spits out the following error message:

╷
│ Error: Invalid index
│
│   on lmhd.me.tf line 348, in resource "aws_route53_record" "lmhd_record":
│  348:   records = yamldecode(file(each.key))["records"]
│     ├────────────────
│     │ each.key is "lmhd.me/widgets.lmhd.me.yaml"
│
│ The given key does not identify an element in this collection value.

Which is okay. You can figure out what’s gone wrong there.

We can’t add custom error messages, because we can’t use validation blocks, as those don’t exist for resources, only variables.

Great! But… Why?

I mean, why go through all this effort, when you could just write the Terraform code?

What’s wrong with this?

resource "aws_route53_record" "widgets-lmhd-me-CNAME-record" {
  zone_id = aws_route53_zone.lmhd-me.zone_id
  name    = "widgets.lmhd.me"
  type    = "CNAME"
  ttl     = "300"

  records = ["lmhd-widgets.netlify.app."]
}

And besides the obvious answer of “I’m doing this because it’s fun…

It actually does have some valid use-cases.

A couple years ago I did a talk at HashiConf about how we generate Terraform code for our Vault configuration.

You can watch it here later:

The short version though… our pipeline generates Terraform code. So for example, if a user wants to add a new policy, they don’t worry about writing the Terraform code for that, they just raise a Pull Request which contains the policy itself, and our pipeline figures out the rest.

That was originally done with some bash scripts… which it turns out, once we had several hundred, was pretty slow.

So we rewrote it in Go and now it’s faster.

But what if we didn’t need any external tools in the first place?

What if we could just have Terraform do all the hard work?

That’s how I’ve been approaching Terraforming my own personal Vault.

So let’s see if I can improve that with some sort of dynamic Terraform like we’ve seen above.

Vault Terraform 2.0

Let’s start with policies, as those are the easiest to conceptualise.

It’s just one file, and you want the content of each of those files to become policies in Vault.

#
# Dynamic Policies from Files
#

locals {
  policy_files = fileset(path.module, "policies/*.hcl")
}

resource "vault_policy" "policies" {
  for_each = local.policy_files

  # Use the name of the policy file as the name of the policy
  name = trimsuffix(basename(each.key), ".hcl")

  # And use the content of the file as the policy itself
  policy = file(each.key)
}

So for an example policy…

  # vault_policy.policies["policies/test.hcl"] will be created
  + resource "vault_policy" "policies" {
      + id     = (known after apply)
      + name   = "test"
      + policy = <<-EOT
            # Test policy
            # Just comments
            # doesn't actually do anything

        EOT
    }

Hey! That looks pretty good 😀

One thing to note though is I’ve had to manually remove the existing policies from Terraform State, so that Terraform doesn’t delete anything. e.g.

$ terraform state rm vault_policy.vault_terraform
Acquiring state lock. This may take a few moments...
Removed vault_policy.vault_terraform
Successfully removed 1 resource instance(s).

$ terraform state rm vault_policy.default
Acquiring state lock. This may take a few moments...
Removed vault_policy.default
Successfully removed 1 resource instance(s).

I could re-import those policies into the Terraform state if I wanted to, so the resulting Terraform plan is a no-op… but as writing Vault policies is idempotent, I’m not really bothered for the moment.

Something I would definitely want to do in a Production environment though.

Let’s load test this for funsies!

As we’ve added more things to our Vault Terraform pipeline at work, the amount of time it takes for the pipeline to run end-to-end is… well, it’s longer than I’d like.

So what if we had like… 10k policies?

How would Terraform Cloud handle that?

Let’s generate a bunch of test policies:

$ for i in $(seq 1 10000); do echo "# test policy ${i}" > test_policy_${i}.hcl; done

Yeah, let’s just casually add about 5MiB to this git repo. No big deal. 😎

Writing objects: 100% (10003/10003), 528.05 KiB | 5.33 MiB/s, done.

Unsurprisingly, simply cloning this chonky boi took Terraform Cloud quite a while.

It wasn’t until about 4 minutes in that it actually started planning anything.

So… maybe I should at least give it a chance with a smaller set first, and go from there.

With a mere 1000 test policies, it took under a minue to Plan, and a similar amount of time to Apply.

Nice.

And adding another 1000?

3 minute plan, 3 minute apply. So 6 minutes end-to-end, with over 2000 resources configured through Terraform.

Pretty good.

And to delete them all once I’m done?

About a minute and a half, end-to-end.

Dang that’s great!

What next?

For me, the next thing I’m going to work on is AppRoles and PKI roles, and then see where we go from there. But that’s a job for another day.

If I were setting up a Vault Terraform pipeline in a new company for the first time, this is definitely the approach I’d want to take.

As much off-the-shelf as possible, and and minimal external dependencies.

And a couple of years ago, I spoke at HashiConf¹ about how we² manage our Vault configuration with Terraform. So it feels like it’s about time I get around to doing something similar for my own Vault.

For now, I’m not gonna do anything fancy; I just want bare minimum Vault Terraformability.

I’ll be using Terraform Cloud for this, because I don’t want to have to do things by hand if I can avoid it, and TFC is cool. Terraform Enterprise also exists, and is almost the same thing as Terraform Cloud, but different in ways I’m not going to get into right now. If you do not have an account, you can sign up for one at app.terraform.io

I’m also not going into how to set up Vault; this is mostly for my own reference, but if anybody else is following this, I’m assuming you’ve got a Vault set up already. If you haven’t got a Vault yet, you may wish to consider HCP Vault, which is now generally available, so you don’t have to put much effort into spinning it up.

My Vault is available over the public Internet directly, because YOLO, so if yours isn’t then you’ll also need to get routing from TFC/TFE to Vault in place.

Let’s begin!

Vault Bootstrapping

Let’s start with the bootstrap configuration on the Vault side.

First thing we need is a Policy, which will grant Terraform the ability to manage stuff. For now, I’m leaving it at the bare minimum, just so I can prove that it’s all working, and I’ll expand on it later (with Terraform itself).

My initial terraform_vault policy looks like this:

# Terraform creates a Child Token to interact with Vault
path "auth/token/create" {
  capabilities = ["update"]
}

# A Test secret, to prove TF is working
# In my case, this is a KVv2 Secret Engine, so we need /data/ in there
path "kv/data/terraform" {
  capabilities = ["create", "update", "read"]
}

Next, we need an Auth method. I’ll be using AppRole for this, because it’s super flexible, and nobody’s written a dedicated Terraform Cloud/Terraform Enterprise Auth plugin for Vault yet³.

Logged in to my Vault UI, I’m going to use the built-in Browser CLI. That’s this little icon in the top right:

I’d love to this in the UI directly… but there’s no AppRole UI yet. Maybe one day.

So, again, bare minimum:

> write auth/approle/role/vault_terraform token_policies=vault_terraform token_ttl=300

Success! Data written to: auth/approle/role/vault_terraform

> read auth/approle/role/vault_terraform

Key                     Value              
bind_secret_id          true               
local_secret_ids        false              
secret_id_bound_cidrs   null               
secret_id_num_uses      0                  
secret_id_ttl           0                  
token_bound_cidrs       []                 
token_explicit_max_ttl  0                  
token_max_ttl           0                  
token_no_default_policy false              
token_num_uses          0                  
token_period            0                  
token_policies          ["vault_terraform"]
token_ttl               300                
token_type              default                

Short TTL, no usage limits or CIDR restrictions, etc. But we’re bootstrapping, and this isn’t Production, so we don’t care right now. Terraform will update it’s own AppRole later to make it more secure.

Terraform Code

Now we need some Terraform code, to actually… you know… do something.

First thing, we tell Terraform we’re using TFC:

terraform {
  backend "remote" {
    organization = "lmhd"

    workspaces {
      name = "vault"
    }
  }
}

This links the repo to a Terraform Cloud workspace (which we will create in a bit), and uses that to store the Terraform state.

Next we need to auth with Vault:

# These variables intentionally left blank
variable "login_approle_role_id" {}
variable "login_approle_secret_id" {}

provider "vault" {
  # Not setting Vault Address
  # We can pull that from the VAULT_ADDR env var

  auth_login {
    path = "auth/approle/login"

    parameters = {
      role_id   = var.login_approle_role_id
      secret_id = var.login_approle_secret_id
    }
  }
}

Here we’re telling Terraform to use AppRole authentication to log in to Vault, and we’re giving it the AppRole’s Role ID and Secret ID as a Terraform Variable.

I’m not specifying where my Vault is; I’ll be doing that with an environment variable.

And then, finally, we’ll have Terraform actually do something. In this case, create a simple KV secret:

resource "vault_generic_secret" "example" {
  path = "kv/terraform"

  data_json = <<EOT
{
  "foo":   "bar",
  "pizza": "cheese"
}
EOT
}

Git Commit, Git Push, Done.

We’ll add a bunch more stuff to this later, but that’s a problem for Future Lucy.

Terraform Cloud

Now we want to set things up in TFC so it applies our Terraform code. There is a Terraform Provider for Terraform Cloud⁴, and I’ll probably look into that in future. For now, I’ll just do things by hand.

First things first, we need to create a new Workspace. I’m using the Version control workflow, so I can link it to a Git repo, and not have to worry about things:

In my case, my Terraform code is stored in GitHub, so I’ll select that:

And I’ll pick my vault_terraform repo:

As far as Settings go, I’m going to call my workspace “vault”:

And I’m going to expand the Advanced options and enable Automatic speculative plans. This should mean that if I make Pull Requests into this repo, then Terraform Cloud will run terraform plan on them. Pretty useful.

We’re almost done!

Terraform Cloud Vault Auth

Now that we have Terraform Cloud configured to use our Terraform code, we need to tell it how to find Vault, and how to auth.

We’ll start with the latter.

Remember how, in the Terraform code, we defined some AppRole variables, but didn’t set any values?

variable "login_approle_role_id" {}
variable "login_approle_secret_id" {}

We’ll set those now.

Back in our Vault UI’s Browser CLI, we can read the AppRole’s Role ID with:

> read auth/approle/role/vault_terraform/role-id

And we can generate a Secret ID with:

> write -force auth/approle/role/vault_terraform/secret-id

In our Terraform Cloud Workspace, we can add these as Terraform Variables. These are secrets, so I recommend marking them as Sensitive, so they do not show up in the UI and cannot be read.

We can also add the address for our Vault as an Environment Variable, VAULT_ADDR.

And that should be everything.

But does it work?

In the Runs tab, we can trigger a new plan:

I’m redirected to the Run details page for this Plan, and pretty quickly I can see that Terraform wants to make some changes:

Terraform v0.14.9
Configuring remote state backend...
Initializing Terraform configuration...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # vault_generic_secret.example will be created
  + resource "vault_generic_secret" "example" {
      + data         = (sensitive value)
      + data_json    = (sensitive value)
      + disable_read = false
      + id           = (known after apply)
      + path         = "kv/terraform"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

I’m asked to confirm before it applies anything, and then we see that the Apply has been successful.

Terraform v0.14.9
vault_generic_secret.example: Creating...
vault_generic_secret.example: Creation complete after 1s [id=kv/terraform]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

And if we check in Vault, we can see that a secret has been created:

And to make sure it’s persisting state and reading from Vault, I’m going modify that secret by hand, then run Terraform again.

This time we see the plan:

Terraform v0.14.9
Configuring remote state backend...
Initializing Terraform configuration...
vault_generic_secret.example: Refreshing state... [id=kv/terraform]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # vault_generic_secret.example will be updated in-place
  ~ resource "vault_generic_secret" "example" {
      ~ data_json    = (sensitive value)
        id           = "kv/terraform"
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

So I’m satisfied that everything is working as it should.

Next Steps

So it’s pretty basic so far, but it’s a good foundation to build from.

My next steps with this will be to Terraform the chicken/egg things. i.e. the vault_terraform Policy and AppRole, and then work my way through the rest of the configuration I already have.

I’m also going to see if I can dynamically determine the IP addresses for Terraform Cloud⁵, and add those as a CIDR restriction on the AppRole.

Should be fun!

I’ve added Netlify CMS to my site, because it’s pretty cool, and seems like a good idea.

The instructions I followed are here: https://www.netlifycms.org/docs/add-to-your-site/

I also used an example config file from here: https://github.com/netlify-templates/jekyll-netlify-cms/blob/master/admin/config.yml

In the case of my blog, that was two commits:

https://github.com/lucymhdavies/lucymhdavies.github.io/commit/2755181bb96d1b19c129257fe513874e802bc82f

This is adds the admin page, Netlify CMS config, and scripts on the main page to redirect me to the CMS once I log in.

I also needed this, because copypaste errors are the best

https://github.com/lucymhdavies/lucymhdavies.github.io/commit/bd48bb649f76c2b2423162873ee669c687a504f4

Does it work?

Yes.

Yes it does. And it’s really nice.

I log into it using Netlify Identity service, using my Google account. I can create a new post, and when I save it it immediately creates a new PR in my GitHub Repo:

https://github.com/lucymhdavies/lucymhdavies.github.io/pull/5

This in turn triggers an automatic deployment, so I can preview my changes.

The only slight annoyance I have with it so far is that my current posts and images are structures like so:

🐳 :lucymhdavies.github.io lucy $ tree _posts/
_posts/
├── 2017
│   ├── 03
│   │   ├── 2017-03-30-Init.md
│   │   └── 2017-03-30-jekyll-from-ios.md
│   └── 04
│       └── 2017-04-02-dns-under-one-roof.md
├── 2018
│   ├── 03
│   │   └── 2018-03-15-nail-polish.md
│   └── 07
│       └── 2018-07-22-emoji-graphs.md
└── cms

7 directories, 5 files

🐳 :lucymhdavies.github.io lucy $ tree images/
images/
├── 404.jpg
├── LMHD_xs.png
├── avatar.jpg
├── bg.jpg
├── both.gif
├── cms
├── post.PNG
├── posts
│   ├── 2018-03-15
│   │   └── captain-obvious.jpg
│   └── 2018-07-22
│       ├── attention.gif
│       ├── emojitracker.png
│       └── final-graph.png
├── ss-color.png
└── ss.png

4 directories, 12 files

And Netlify CMS doesn’t appear to have an obvious way of nested subdirectories for images and posts, meaning it doesn’t automatically see my previous posts.

No matter. That’s easily fixed. I moved my posts to a common directory:

https://github.com/lucymhdavies/lucymhdavies.github.io/commit/2047f087a2985a1f732dc1bd45bdd81817a8b7d0

With that change, Netlify CMS picked up my existing posts

I’ve not done this for images yet, as I’d need to go through all my existing posts and update them inline too. I might do that at some point, but I’m not too worried for now.

But all-in-all… Seems pretty cool so far :)

Slides

https://talks.lmhd.me/silly.html

Script / Speaker Notes

0 - Title slide

<show emoji and twitter handle>

As much as I’d like my talk to be named Upside Down Smiley Face Emoji, you’d have no idea what to expect seeing that on meetup.com

<reveal title>

So I’m gonna talk about making silly things, and why you should do it.

1 - Emojis - Example of being Silly

Let’s start with an example. I have a reputation for perhaps being a little enthusiastic when it comes to emojis. And so while ago, this manifested as this:

<screenshot of emoji grafana dashboard>

• This is a screenshot of something I put together one Friday afternoon at work.

<live view>

• And here it is live.

• What you’re looking at is a graph of emoji usage on Twitter, in close to real-time. I’ve filtered it to one unpopular one in particular. If the demo gods will allow, here’s a tweet that proves it.

• So this is built on top of something called emojitracker, which uses the twitter apis to keep track of all tweets containing emojis, in real-time. And it has its own streaming api, which returns a stream of linebreak-separated-json with stats about what emojis have been used recently.

• The part I wrote takes that data and outputs a Prometheus metrics page, which I’m showing here as Grafana graph. So this graph is 3 or 4 APIs abstracted from what’s going on on twitter.

• If you’re unfamiliar with Prometheus, it’s a really good tool for monitoring your systems. Usually used for things like how much CPU idle a particular server has, or memory usage, disk space. It has a lot of integration, but you can write your own custom integrations with their go SDKs, which is what this is.

And you may be asking yourself… why?

• And I made this for two reasons:

Firstly, Just Because.

This silly idea of mine was prompted by a twitter bot I found which was tracking the least used emojis in emojitracker, tweeting it out on a schedule.

• The only thing even resembling a purpose for it was as part of an experiment, to test a theory I had, to see if I could predict what people would do in response to this bot. I suspected that, with this bot getting attention, that would be enough that what was the least used at the time would eventually become the second least used.

Crossover point

• And that allowed me to get this graph, which is the moment that happened, about 2 days after I created this, about 12 hours later than I predicted based on linear regression.

• And it didn’t actually take that long to put together either. I said an afternoon, but it’s building on a similar thing I put together a few weeks prior which was getting stats from twitch.tv, so I’d already done the hard work. I’m not gonna show the code on screen, because it’s not that interesting, and not actually very good. But you can dig it up from my GitHub later if you want.

• But both this, and the original Twitch version, gave me a much better understanding of how Prometheus exporters work, which meant that a few weeks later, when I needed to do one for my job, it wasn’t a strange new concept that I’d have to learn.

2 - So why am I talking about this?

First we have to go back a few years…

3 - Rewind

• Back when I was about yay tall, went by a different name, and thought Visual Basic 6 was the coolest thing ever, I’d make all sorts of random stuff for fun. I didn’t really know what I was doing, and was blissfully ignorant of that fact; I was just making stuff, because I had ideas in my head I needed to get down on silicon.

• What happened?

• It wasn’t until I got to university that I was actually properly taught how to write code. There was suddenly this idea that there was a right or a wrong way of doing things, and that my code could be bad. It didn’t help that I was surrounded by smart people who were better than me (my brain conveniently filtered out the people who weren’t).

• My skills improved, and my code got much better, as I learned better ways of doing things, but I also got to see and appreciate what quality looked like… and there was a gap between that and my work. I wasn’t there yet, so I’d feel like a failure.

4 - Impostor Syndrome - The problem

Impostor syndrome begins

• This is impostor syndrome, a feeling that you don’t actually know what you’re doing, despite evidence of the contrary. Apparently about 70% of people get it at some point in their lives.

• It got worse for me when I started my first job, so in addition to being surrounded my smart people, I was surrounded by people who understood the massive million-lines-of-code monoliths the company developed. Let’s ignore the fact that no one person really understood the whole thing.

• It was exhausting, and the side projects I was working on at home stalled.

Fear of harsh judgement

• So part of this comes from a fear of being judged harshly. That people will look at your work and mock it for being so bad.

Fear of harsh judgement, cont.

• This fear of judgement in my case is not just limited to coding, but is sort of a horizontal slice through all aspects of my life.

• For example, I’m a woman, and a trans woman at that, I’d find myself sometimes not fitting in with the heavily male dominated industry. I tried to fit in, so that I wouldn’t be judged for being different, but I ended up spending energy on that which could have been better spent elsewhere.

• <wonder women who go> <trans gopher>

Fear of Failure

• This is something I struggled with for the longest time; when faced with a new technology I need to use for my day job, I find it hard to admit that learning something new under time pressure can be stressful, and take longer than anticipated.

• This often leads to me staying late in office if I don’t make as much progress as I had hoped, which is something I’m working on getting better at.

• Something I realised was happening with with my side projects was that I wasn’t working on things because I expected them to fail. I’d cling on to ideas as “something I’d work on when I’m better at this”, because I didn’t want them to fail. But of course, that meant I wasn’t making them.

Big Ideas are overwhelming

• There’s this idea I first learned about in the context of art, which is that as you become a better artist, you get better at appreciating quality faster than your ability to create improves. So you feel like you’re getting worse, when what’s actually happening is your goals are getting bigger.

• That’s something which definitely happened to me; I was that kid in uni who thought they were gonna create the next big social network that everyone would want to use… based around an idea which, in retrospect, is actually maybe only useful for me and a few other people, and which could probably be achieved with something much simpler than what I had in mind.
But I didn’t know that back then, so I’d accumulate feature creep over time without so much as writing one line of code.


5 - The Solution?

These days, I have the word “Senior” in my job title, so people seem to think I’m awesome. They might be right… [pause for laughter?] but nah, the impostor syndrome is still there. I’m just getting better at dealing with it.

Doing Stuff Because It Scares You

• So this goes back to fear of being judged. Turns out, asking people to review my code… not that big a deal. Everywhere I’ve worked so far, any time I submit things for review, I get constructive feedback. Yes there are often problems with the code, but it’s clear that those are not problems with me, and I always learn how to do things better in future.

• I’ve taught people how stuff works, both at my old job and my current job. That was a weird feeling at first: impostor syndrome telling me I didn’t know what I was talking about, and me proving it wrong by explaining stuff to people.

• And of course, I’m standing in front of you giving this talk, despite the fact that just last night I was convinced I had no idea what I was talking about.

• Doing scary things makes doing scary things easier.

• And over time I’ve learned to be myself more, and I’ve found myself not worrying so much about what others think of me.

Prototypes & Silly Things

• Big ideas can be great, but they’re made up of lots of small bits, and unless you’re a super genius you won’t immediately know how to make the entire thing in one go. So biting off small bits, experimenting with ideas related to it, and building up your knowledge, until eventually you do have all the skills you need to build something big if you want to.

• This should be obvious, but it’s something I used to undervalue.

• Experiment. Prototype stuff. Try lots of things. If one works out well, great, you can improve on it.

• Often when you have an idea, you have no idea if it’s any good, so you can spend hours or weeks or months over engineering the most elegant of things… only to find that actually, it wasn’t that great an idea in the first place.

• I have a git repo full of unfinished experiments, where I wanted to try out a new library or an idea I had.

• Some of those ideas have graduated to their own repos, like the emoji exporter. Some are still little things I’m tinkering with every now and then. None of them are examples of my best code; all are ideas I wanted to try out.

• In the spirit of learning not to be afraid of writing bad code, I keep it open. I don’t imagine many people are going to find it, but it’s there should I ever want to point somebody at a thing I’ve been experimenting with.

L&D Time

• You may remember I said this emoji exporter was done on a Friday afternoon at work.

• I wasn’t slacking off, honest; we have a thing called Learning & Development time where we can work on stuff that’s not work related, or read a book, watch training videos, etc. Anything to further our own learning and development.

• This is something I’ve seen a few companies do. For me personally, I get silly ideas like that Prometheus exporter for twitter emojis all the time, and before L&D time was a thing, I’d make a note of them, but never really find time outside of work to focus on them.

• And one of the great things about this is that it teaches you to give yourself Permission to Fail.

• L&D time is for you to do what you want, you’re not committing to shipping anything at the end of it. With this structured L&D time, or even in your own time making something silly without any commitment to producing anything of value at the end of it, you’re training yourself to try new ideas, and accept that sometimes stuff doesn’t work out. Then in theory, when you’re struggling with your day job, it becomes less difficult to admit that it’s not working out and that you might need help. I’m still working on that one, but this sort of thing really helps with that.

Go!

• And I want to bring things back to Go; this is Go Sheffield after all.

• While Ruby was the language that got me back into working in side-projects, it was Go that made things fun again.

• There are several thing which really drew me to Go which I want to highlight

• Superficially, it’s impossible to write badly formatted Go code when GoFmt and GoImports exists, and can be configured to run automatically when you save. I know this isn’t unique to go. Saves me from being distracted, and just get on with writing the code.

• It’s also really easy to read and understand, especially when it come to figuring out what comes from which libraries. This sort of thing makes it much easier to teach yourself the language than, say, Ruby, where I’ve often seen package names that don’t quite match the name of the gem which provides them, and monkeypatched functions that you just have to know where things are coming from.

• But I think mostly, the reason is that when I first started learning go, like when I was learning ruby, and like when I was learning Visual Basic 6 back in the day… I was learning by myself how it worked rather than being taught. And it seemed super intimidating at first (my first Go project was this tool that someone in my team wrote that was complicated and intimidated me. He’s now gone off and is doing other things, and now I’m the expert in this particular tool).

• I can look back and see how far I’ve come with it, so when I look forward at all the things i know exist but which I’ve not used before, i know that I can teach myself how to do anything I want with it.

• And that even if I don’t have an obvious use for some features now, that I’ll be able to come up with something silly I can use to teach myself how it works.

6 - End

So yeah, that’s all I have for you.

Look for things you don’t understand, and come up with silly ideas as an excuse to experiment with them.

Thank you!

Initial Interest

A few days ago (the day after World Emoji Day as it happens) I discovered a tweet:

Y’all know what to do https://t.co/YCRHtJfWAk

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

Apparently there’s a bot keeping track of which emojis get the most use. It’s made by Jeremy Schmidt and is called, fittingly, LeastUsedEmojiBot. You can find the code on GitHub

As a fan of emojis in general, this got me interested. Obviously I wanted to try to help the humble Aerial Tramway emoji reach its true potential of second least used emoji on Twitter.

As

To keep track, we are on 128916 at the moment. 🚡

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

You

And I think it only counts number of tweets it’s used in, rather than number of uses.

It does appear to update pretty dam quick.

🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

May

Yep. One per tweet. 🚡

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

Have

Also, nice.

🚡 https://t.co/gE9CpRsIdb

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

Noticed

Me: *slaps twitter*

This baby can hold so many 🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡🚡

— Lucy Davinhart (@LucyDavinhart) 18 July 2018

Digging into this further, I found that the bot got its data from a site called Emojitracker, made by Matthew Rothenberg

This site gets realtime updates from the Twitter Streaming API of all emojis used on Twitter.

That’s a LOT of data, and some nice APIs too:

There’s a REST API, to get a snapshot of all emojis, and a Streaming API for updates.

Now I was even more interested.

Writing a Prometheus Exporter

It was at this point that I noticed that usage of the Aerial Tramway emoji was increasing faster than its rival, the Input Symbol for Latin Capital Letters. This was almost certainly due to LeastUsedEmojiBot highlighting it. At some point it would overtake, but I wasn’t sure how soon.

It was now Friday afternoon. At work, we have a thing called “Learning and Development Time”, in which you can (within reason) basically do whatever you like to further your personal development. It doesn’t even have to be work related. In the past, I’ve used this time to work on various personal dev projects, which I may blog about at some point.

One of those previous projects was a Prometheus Exporter for Twitch.tv. My wife is a Twitch streamer (obligatory plug: SeraphimKimiko), and I’m a nerd, so I wanted to keep track of how many people were watching her live. So I made this, written in Go, consuming the Twitch APIs, and exporting them as Prometheus metrics. I included a Docker Compose file to spin up the Prometheus exporter, a Prometheus server to scrape it, and a pre-configured Grafana instance to draw pretty pretty graphs. And it’s written in Go. Because of course it is.

Thanks to that project, I had most of the code already to graph data from the Emojitracker APIs. I got to work on what would eventually become my Prometheus Exporter for Twitter Emojis.

Let’s see what the Emojitracker API gives us. The API endpoint I’m interested in is https://api.emojitracker.com/v1/rankings, which returns JSON like:

$ curl -s https://api.emojitracker.com/v1/rankings | jq .
[

...

  {
    "char": "🚡",
    "id": "1F6A1",
    "name": "AERIAL TRAMWAY",
    "score": 130982
  },
  {
    "char": "🔠",
    "id": "1F520",
    "name": "INPUT SYMBOL FOR LATIN CAPITAL LETTERS",
    "score": 130893
  }
]

First, we need a Prometheus metric. I used a Gauge (which, in Prometheus is a metric which can go up or down). Arguably I should have used a Counter (which can only go up), but this was a proof of concept, and I wasn’t sure what happens if tweets get deleted. I’m interested in the emoji itself (because apparently both Prometheus and Grafana support those just fine), as well as some plaintext identifiers:

emojiScore = prometheus.NewGaugeVec(
  prometheus.GaugeOpts{
    Namespace: "lmhd",
    Subsystem: "emoji",
    Name:      "twitter_ranking",
    Help:      "Number of uses of this emoji on twitter",
  },
  []string{
    // Which emoji?
    "emoji",
    "name",
    "id",
  },
)

Now we need to populate that with some data.

I used json-to-go to quickly generate a type which matched the output of the API:

type EmojiRankingsResponse []struct {
  Char  string `json:"char"`
  ID    string `json:"id"`
  Name  string `json:"name"`
  Score int    `json:"score"`
}

For my Twitch exporter, I had used curl-to-go to generate some Go code to call the APIs, and return structs. The code my Emoji exporter used was based off that.

There are two functions here. The first calls the API, and returns (among other things) the response body:

func EmojiRankingsRequest() ([]byte, *http.Response, error) {
  // Modified from code generated by curl-to-Go: https://mholt.github.io/curl-to-go

  url := "https://api.emojitracker.com/v1/rankings"

  req, err := http.NewRequest("GET", url, nil)
  if err != nil {
    log.WithFields(log.Fields{"url": url}).Errorf("%s", err)
    return []byte{}, nil, err
  }

  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    log.WithFields(log.Fields{"url": url}).Errorf("%s", err)
    return []byte{}, resp, err
  }
  defer resp.Body.Close()

  body, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    log.WithFields(log.Fields{"url": url}).Errorf("%s", err)
    return []byte{}, resp, err
  }

  return body, resp, nil
}

The second takes that response and converts it into something of type EmojiRankingsResponse.

func Rankings() (EmojiRankingsResponse, error) {

  // init an empty response
  response := EmojiRankingsResponse{}

  // body, resp, err
  body, resp, err := EmojiRankingsRequest()
  if err != nil {
    log.Errorf("%s", err)
    return response, err
  }
  if resp.StatusCode != 200 {
    log.Errorf("Error code %s, Error: %s", resp.StatusCode, err)
    return response, err
  }

  err = json.Unmarshal(body, &response)
  if err != nil {
    log.Errorf("%s", err)
    return response, err
  }

  return response, nil
}

You can find this in emoji.go

So with that in place, I can populate my Prometheus metrics. In my main.go, I iterate through all emojis in that response, and update their corresponding Prometheus metric:

// Init with rest API
rankings, err := Rankings()
if err != nil {
  log.Fatalf("%s", err)
}

for _, emoji := range rankings {
  emojiScore.With(prometheus.Labels{
    "emoji": emoji.Char,
    "name":  emoji.Name,
    "id":    emoji.ID,
  }).Set(float64(emoji.Score))
}

This worked great! I now had some metrics!

$ curl -s http://localhost:8080/metrics | grep -i strawberry
lmhd_emoji_twitter_ranking{emoji="🍓",id="1F353",name="STRAWBERRY"} 9.273592e+06

But these were static, which is not much use to me. I needed updates.

As a proof of concept, I initially just called the REST API every minute for updates, and updated the prometheus metrics accordingly. But this was me being lazy. The REST API Documentation says you should not do this:

When to use the REST API

In general, use the REST API to build an initial snapshot state for a page (or get a one-time use data grab), but then use the [Streaming API][https://github.com/emojitracker/emojitrack-streamer-spec] to keep it up to date.

Do not repeatedly poll the REST API. It is intentionally aggressively cached in such a way to discourage this, in that the scores will only update at a lower rate (a few times per minute), meaning you have to use the Streaming API to get fast realtime data updates.

🚨 IN OTHER WORDS, IF YOU ARE POLLING FREQUENTLY FOR UPDATES, YOU ARE DOING SOMETHING WRONG AND YOU ARE A BAD PERSON. 🚨

(Note that this is a design decision, not a server performance issue.)

I’d never used a streaming API before, so didn’t know what to expect.

According to the documentation, I could expect:

a JSON blob every 17ms (1/60th of a second) containing the unicode IDs that have incremented and the amount they have incremented during that period.

Example:

data:{'1F4C2':2,'2665':3,'2664':1,'1F65C':1}

I curl’d the API, to see what this looks like, and wow that updates quick!

Looks a bit like this:

$ curl -s https://stream.emojitracker.com/subscribe/eps
data:{"1F405":1,"1F60C":1}

data:{"1F450":1,"1F493":1,"1F498":1,"1F602":1,"1F60D":1,"1F629":1,"25B6":1,"26BD":1}

data:{"1F64F":1}

data:{"1F60F":1,"267B":1}

data:{"1F308":1,"1F4F2":1,"1F602":2,"1F61C":1,"1F64B":1,"2B50":1}

data:{"1F607":1}

data:{"1F335":1,"1F3A5":1,"1F447":1,"1F4F2":1,"1F51E":1,"263A":1,"2705":1}

data:{"1F602":2,"2764":1}

data:{"1F621":1}

data:{"1F48F":1,"1F602":1}

So I needed to consume that URL, look for lines beginning with data:, and parse the JSON into something useful.

First thing was to just keep reading the API:

resp, _ := http.Get("https://stream.emojitracker.com/subscribe/eps")

reader := bufio.NewReader(resp.Body)
for {
  line, _ := reader.ReadBytes('\n')
  lineString := string(line)
  
...
  
}

We only care about lines which begin with data:, so let’s get those (and drop the data: prefix):

// Lines look like
// data:{"1F449":1,"1F44D":1,"1F60F":1,"26F3":1}

if strings.HasPrefix(lineString, "data:") {

  data := []byte(strings.TrimPrefix(lineString, "data:"))
  
  ...
  
}

The JSON itself is a series of string keys, with integer values. In Go that could be represented as: map[string]int.

I wasn’t sure if Go would let me parse the JSON directly into something like that, but I gave it a try:

jsonMap := make(map[string]int)
err = json.Unmarshal(data, &jsonMap)
if err != nil {
  panic(err)
}

Sure enough, it worked! It might error at some point, but like I say, proof of concept.

All that was left was to update my metrics. I used the rankings object I created earlier to lookup the name and emoji for the ID, and used that to increment my prometheus metric:

for key, val := range jsonMap {
  for _, emoji := range rankings {
    if emoji.ID == key {
      emojiScore.With(prometheus.Labels{
        "emoji": emoji.Char,
        "name":  emoji.Name,
        "id":    emoji.ID,
      }).Add(float64(val))
      log.Debugf("Char: %s (%s) : %d", key, emoji.Name, val)
    }
  }
}

And that’s basically it. It could absolutely do with some tidyup (for example, being able to lookup the emoji details from the ID, without having to iterate over rankings), but it works fine for my proof of concept.

Now, let’s get this into pretty pretty graphs.

Pretty Pretty Graphs

I went through a few iterations of this, before I settled on one I liked:

Potentially relevant to @nocturnalBadger's interests.

🚡 pic.twitter.com/4ecnaHSd9p

— Lucy Davinhart (@LucyDavinhart) 21 July 2018

I was predominantly interested in the bottom two emojis, so my dashboard kept track of those two.

I had the overall usage in a Graph panel.

This used Prometheus’ “Bottom K” operator, which I used to filter out only the bottom 10 metrics):

bottomk(10,lmhd_emoji_twitter_ranking)

I also had indvidual Singlestat panels for the two emojis, configured for example with:

lmhd_emoji_twitter_ranking{emoji="🚡"}

I left this running overnight to gather some data, then woke up this morning to discover that, oh no! Disaster struck!

Turns out, at some point in the night my prometheus exporter had stopped consuming the streaming API!

time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F694 (ONCOMING POLICE CAR) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 203C (DOUBLE EXCLAMATION MARK) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 267B (BLACK UNIVERSAL RECYCLING SYMBOL) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 2764 (HEAVY BLACK HEART) : 2"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F618 (FACE THROWING A KISS) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F629 (WEARY FACE) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F494 (BROKEN HEART) : 1"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F602 (FACE WITH TEARS OF JOY) : 2"
time="2018-07-22T01:43:49Z" level=debug msg="Char: 1F614 (PENSIVE FACE) : 1"

My numbers were stale!

Fortunately, 🚡 had not yet overtaken 🔠, so there was still time for me to see it happen.

One quick docker restart emoji_exporter_emoji_exporter_1 and we were collecting data again.

Gave my Prometheus exporter a restart, and new data is coming in.

🚡 has certainly made quite a bit of progress since this morning, but the other one has too.

They're close. Shouldn't be long now. pic.twitter.com/zTuUhuRe3n

— Lucy Davinhart (@LucyDavinhart) 22 July 2018

I kept watch, and just 20 minutes later, we did it!

So, now that 🚡 is the second least used emoji on Twitter, I'm not sure what will happen now.

Some people will continue to tweet, as @leastUsedEmoji will not have updated yet. I believe it's scheduled to update within 25m.

At that point... I'm expecting a massive spike. pic.twitter.com/nTreFd95BH

— Lucy Davinhart (@LucyDavinhart) 22 July 2018

This is the tweet which pushed 🚡 into second least used!

YAY! https://t.co/iVhWOjnGoY

— Lucy Davinhart (@LucyDavinhart) 22 July 2018

Celebrations all round!

I made a couple of tweaks to the dashboard following that. The new version includes a dropdown, so you can select which emojis you want to compare (from all of them); a table, showing specifically the bottom 5 emojis; and a rate graph, showing how many individual tweets there were over a time interval of 10 minutes.

I've updated the grafana dashboard slightly, if anyone is still interested.
🚡

Now includes a table

Code:https://t.co/2PwmnRIC6y

Interactive Snapshot:https://t.co/OT3sNW0jnY

Screenshot: pic.twitter.com/6R01KfoAQC

— Lucy Davinhart (@LucyDavinhart) 22 July 2018

I've also added a rate graph, which converts the cumulative number of tweets into tweets over time (i'm using a 10 minute interval).

Usage spikes are even more obvious in this graph. pic.twitter.com/ocbWwFtbAD

— Lucy Davinhart (@LucyDavinhart) 22 July 2018

What was the point? What did I learn?

You mean I need to do stuff because it has a point?

Nah.

That’s not a thing.

Seriously though, this was a fun thing to work on, especially as I was able to re-use so much code, letting me play a bit more without figuring out how to just get something working.

I already do a lot of realtime monitoring of a bunch of stuff at work, to make sure I don’t get woken up in the middle of the night (or to ensure I definitely do, if something needs fixing). But these two things (my Twitch exporter, and my Emoji exporter) include monitoring of external APIs, and human nature.

This one in particular was fascinating. Because it was a relatively small dataset (the LeastUsedEmojiBot only has 14k followers on Twitter), I could clearly see cause and effect. For example, the spike in usage following the bot’s announcement that 🔠 was now the least used.

It was also interesting being able to make preditions, using Prometheus’ predict_linear() function:

According to my grafana dashboard, we should be overtaking Truncated Latin Alphabet some point early tomorrow. 🚡 https://t.co/w63l6lRJDv

— Lucy Davinhart (@LucyDavinhart) 21 July 2018

Using Prometheus's built in predict_linear() function, the two emoji should be approximately equal in about 9 hours (5am for me in the UK). 🚡

— Lucy Davinhart (@LucyDavinhart) 21 July 2018

Specifically by 5amhttps://t.co/RvULAuVx7y

— Lucy Davinhart (@LucyDavinhart) 21 July 2018

I was wrong, of course. Human nature is not so easily predictable by simple linear regression.

But yes. This was fun. I need to do silly things like this more often.

I’ll leave you with this video, by a very inspiring woman:

And one more graph (click on it to go to the interactive version!):

About a month ago, I saw and retweeted this thread (ThreadReader link to the full thread) about femininity in the workplace.

Interesting thread.

Manages to express in words something I’ve been struggling to understand about myself recently, w.r.t. (my lack of) femininity in the workplace.

(And it’s made even more complicated when you’re enby) https://t.co/ttDScd3zgf

— Lucy Davinhart (@LucyDavinhart) 19 February 2018

Why yes the largest section in the tech business article I'm writing right now is Fashion Tips

— Stephanie Hurlburt 🔜 GDC (@sehurlburt) 19 February 2018

And it got me thinking about how, for the most part, I still present to the world about the same as I did before I transitioned. i.e. jeans, nerdy tshirt, usually a hoodie, and no make-up at all. My hair is much longer than it used to be, and I have rather obvious boobs now, but for the most part… yeah. Not much has changed.

I tell myself that this is just my aesthetic, and not to worry about it. And because I also consider myself to be non-binary¹, I kinda just lumped it in with that.

But then, a couple of weeks ago, I was in London with a friend of mine, and we happened across a Belgian shop which, among many other things, had a large selection of nail polish. It reminded me that I’ve wanted to paint my nails basically forever, but never really got around to doing it. So I bought some. Specifically, I bought some in what was the closest approximation I could find to LucyPurple

And now I’ve found nail polish in #LucyPurple with the help of my local nail polish consultant, @MaartjeME pic.twitter.com/eXZcmXp6os

— Lucy Davinhart (@LucyDavinhart) 3 March 2018

But then… they just sat in the bag, unopened.

I told myself that I just hadn’t got around to it yet, or that I didn’t have time, or that I needed extra tools ². But these were just excuses I told myself.

This was looking like it was going to be a repeat of an idea I came up with after FOSDEM. In that case, every day of the long-weekend, I was almost exclusively in the company of awesome queer folk. So I let myself wear ever so slightly more feminine clothing (including an awesome trans-gopher shirt on the Saturday).

After I got back home³, I came up with the idea of a “Femme Friday”, whereby I’d let myself wear some of my more feminine clothes to work. The furthest that ever went was wearing a long plaid shirt on top of what I normally wore to work.

And then, a few days ago I saw and retweeted this:

Cis women, if you ever see a trans woman in her 30s, 40s etc. wearing something that looks too juvenile for her, leave her alone about it. She literally never got to be the age you were when you wore it. Let her have this, please.

— Faith Naff 🏳️‍🌈🌹🦋 (@FaithNaff) 12 March 2018

And I think that’s when it clicked in my head what was actually going on here, and it’s something which should not have been a surprise to me:

I’m self-conscious.

As much as I like to tell myself that I don’t care what other people think of me, it occurred to me that I’m still scared to do much in public that’s actively feminine.

On the far extreme of this, I have several skirts, which I desperately want to wear outside, or to work, but I’m too scared to actually leave the house in.

But that’s silly!

I’ve been self-conscious about a great many things before, and I’ve got over it!

I used to be terrified of using female toilets in public, but now it’s no big deal.

I used to be terrified of talking to my team’s client at my previous job, but that became no big deal fairly quick.

I used to be terrified of public speaking (I still am, to an extent), and then I did a talk in front of about 200 people at FOSDEM, and it went okay, so now I’m considering doing more of that kind of thing.

My point is… I do a thing which scares me, and then it’s fine.

So perhaps I should do more things that scare me?

So, long story short, I painted my nails last night.

It wasn’t perfect (I wasn’t expecting it to be), but it looks okay. And importantly, I’ve come in to work and it’s no big deal.

I found myself hiding my nails on the tram in to work this morning (initially), and hiding them at work (initially). But now… it’s no big deal.

So I should just do more stuff like this which scares me.

It’ll be fine, and I’ll feel much better for doing it⁴.

#LucyPurple update pic.twitter.com/BIjDN5GHhf

— Lucy Davinhart (@LucyDavinhart) 15 March 2018

• Queer Code Yorkshire, January 2018
• FOSDEM, February 2018
  • See also: my write-up of the event on the SB&G Technology blog: FOSDEM 2018

Video

Slides

Google Drive Slides

Speaker notes embedded in the Google Slides

The FOSDEM talk page has links to code and asciinema recordings

The vast majority of my collection I’ve purchased from GoDaddy. I’m fine with that. I often hear bad stuff about them, but they’ve been good to me so I’m in no rush to move.

But the DNS part… that’s spread out all over the place. Some of it is managed by GoDaddy, some of it is delegated to other hosting providers¹. lmhd.me specifically is on PointDNS².

So I’m going to move it all to one place as much as possible: AWS Route53

Why Route 53?

There are lots of DNS solutions out there: some free, some paid, some self-hosted. I could have gone with any of them.

But I wanted something I could automate, preferably with Terraform (because I’m familiar with that from work).

Terraform has many DNS providers to chose from, and I looked through quite a few of them.

But I eventually decided on Route53, partly because it’s cheap³, and partly because I’m thinking of moving some of my Heroku stuff over to AWS at some point.

ALIAS records for my apex domain

The reason I went with PointDNS origiainally was because my apps were on Heroku, and it was available as a free add-on.

The original lmhd.me was a static page on HostGator, then I moved it to Heroku. Now it’s on GitHub Pages.

When it was on Heroku, I needed what PointDNS (and others) refer to as an ALIAS record. From the PointDNS console:

Please specify a DNS name to alias. Point will automatically duplicate A and AAAA records from this address at 15 minute intervals. This may be used as an alternative to a CNAME for the root of a domain.

In the case of lmhd.me this works like:

• lmhd.me is an ALIAS record for lucymhdavies.github.io
• lucymhdavies.github.io is a CNAME for github.map.fastly.net
• github.map.fastly.net has A records for the relevant IP addresses
• PointDNS creates a A records for lmhd.me based on the github.map.fastly.net A records

Route 53 has no such concept. The only Alias records it has is for other AWS services.

I was originally thinking of just setting up A records, as GitHub suggests, but I can do better than that.

Terraform can itself query DNS, e.g. CNAME records and A records.

So what I ended up doing was the following:

#
# lmhd.me apex domain --> github pages
#

# Lookup CNAME for lucymhdavies.github.io.
# Result is stored in the variable:
# data.dns_cname_record_set.lucymhdavies-github-io.cname
data "dns_cname_record_set" "lucymhdavies-github-io" {
	host = "lucymhdavies.github.io"
}

# Get IPs for that CNAME
# Result is stored in the array:
# data.dns_a_record_set.github-io.addrs
data "dns_a_record_set" "github-io" {
	host = "${data.dns_cname_record_set.lucymhdavies-github-io.cname}"
}

# Set up the A records for lmhd.me
resource "aws_route53_record" "lmhd-me-A-record" {
	zone_id = "${aws_route53_zone.lmhd-me.zone_id}"
	name    = "lmhd.me"
	type    = "A"
	ttl     = "60"

	records = [ "${data.dns_a_record_set.github-io.addrs}" ]
}

This provides me with a good middle ground between simply setting the A records manually, and setting up something (a Docker container?) to replicate the functionality PointDNS provides.

What next?

I’m not done yet. I have more domains to move, but they’re all CNAME and A records, so nothing too interesting with that. I just need to get around to doing it.

The only manual part of the process is updating GoDaddy’s nameservers for the domain to point to Route53.

There’s a Terraform plugin I could use for that, and there are other tools which use the GoDaddy API, for example a Node.js script

But given that it’s a one-off thing per domain, and I don’t have too many domains, I’m not too bothered right now.

Another option for me to consider is that domains from Amazon cost about the same as I’m paying from GoDaddy. When that domain is up for renewal in January, I might consider that.

At some point in future I’m going to have Terraform Plan run automatically, and send me a Slack notification if there are any changes to be applied⁴. But one step at a time, eh?

I need to figure something out.

This post was written entirely on my iPad and iPhone, with a bunch of different apps, in my attempt to figure out how best to write posts when away from my laptop.

Prose.io - Free

Prose.io seems good. This was one of the first things I found when I Googled for Jekyll editors, so I think it is designed specifically for Jekyll sites. I have been writing this post in that site so far, and it works okay, but not particilarly well.

The most annoying part is that it does not take advantage of iOS’s autocorrect¹, e.g. spelling, uppercasing first letters, double-tap space for full stop.

Plus it’s a web app, so I need Internet access to use it².

But it does integrate directly with GitHub, so I save posts and drafts easily.

So while this does kinda work, it’s not great, and I can see myself getting frustrated with it quite easily.

So. Let’s find other options.

Octopage - £0.99

Octopage on App Store

This app actually works quite well. It has offline editing, preview, and a Markdown syntax guide.

There are a few bugs though, the most annoying of which is that it occasionally scrolls up when I preview the post then go back to edit mode.

It also only has drafts local to the app. No way (yet) of using the drafts directory in the repo. I’ve asked the dev to add that functionality, so we’ll see what happens.

That’s not a deal breaker, but it does mean that any post I don’t complete in this app can’t be picked up on my laptop.

Which means that, while I can write my thoughts on the app using the app itself, I need to update my page draft with a different app, so back to Prose.io for now.

Source for iOS - Free?

Source on App Store

The app is actually free, but being able to push back to GitHub is an in-app purchase. Sort of.

This one took a while to get started. It wasn’t able to set up my SSH key to GitHub, so I had to add that manually. Once that was in place, it all went swimmingly.

This one is a full fledged git client, so it lets me edit any file in the repo. And it also has offline editing, which is nice.

No preview functionality, but that’s fine; that’s not what the app is for.

It does have a few keyboard issues though. It has its own keyboard, which can be used in other apps. It looks like it is supposed to have quick access to special characters, but I can’t see how that is supposed to work. You can disable the custom keyboard, but even then there is none of iOS’s autocorrection, which makes it frustrating to use.

I would like to love this app, but I would need more keyboard behaviour settings before I am happy with it.

I have listed it as free with a question mark because it says that pushing to a remote repo is part of the £4.99 in app purchase, but it looks like that works fine in the free version. I suspect the answer is that pushing to other remote repos is the premium feature. Or it could be a bug, because when I try to push without committing at the same time, it prompts me to pay for then in app purchase.

Git2Go - Free

Git2Go on App Store

Another full git client. Without looking at it too much, it doesn’t appear to have all the git functionality of Source, but I don’t need it to.

This one just has the iOS default keyboard, and again the autocorrect functionality is nowhere to be seen. But I can send feedback from directly within the app, so I’ve sent the suggestion to the dev.

This app is definitely free though, if you don’t need private repos or GitHub Enterprise, so that’s a bonus.

It took me a while to figure out, but you can move files, which means you can publish posts. Which is nice. Looking back on Source, it can’t do that.

It also lets me view diffs before committing, whereas Source would only let me see which files have changed, but not the diff.

I don’t anticipate needing to post many images, but that is possible in this app, where it isn’t in Source.

So… what will I use?

There are a few more apps I could try, but at this point, as long as Git2Go’s devs allow you to toggle autocorrection, I can definitely see myself using it.

This was really the only problem with Jekyll that I needed to figure out before being satisfied that it was a good blogging platform for me (as opposed to Wordpress, for example).