Skip to content
Gateway Arch

My Hack Day experience with API Gateway and Apache Velocity

Published by Martin Smith on June 25, 2017

During my last visit to the Castle, I had the opportunity to join a hack day event with our awesome Solutions Architects at Fanatical Support for AWS at Rackspace. While my team does try to do hack day events every few months, I was excited to join a totally different group with different skillsets and work with them more closely than I had before. I opt’d for the group who wanted to build a specific project (details omitted) using API Gateway, Elasticsearch, Lambda, and Slack.

Specifically, here’s the functionality we built:

  • register an outgoing webhook that posts some Slack data to API Gateway when a trigger word is spoken
  • define an API Gateway endpoint that transforms the request into an Elasticsearch query
  • create and populate data into a hosted Elasticsearch cluster that runs the query
  • a second response transformation that converts the results back to a form acceptable to Slack
  • returns the response back to Slack as a response to the original webhook

This is actually a pretty common design. Since I hadn’t used API Gateway very much at all, I started on that part… we theorized we might be able to skip the Lambda requirement entirely if we could transform the requests and responses from Slack into forms acceptable to Elasticsearch. This is where I discovered the Apache Velocity template language.

What’s an API Gateway integration?

In API Gateway, an API’s method request can take a payload in a different format from the corresponding integration request payload, as required in the backend. Similarly, the backend may return an integration response payload different from the method response payload, as expected by the frontend. API Gateway lets you use mapping templates to map the payload from a method request to the corresponding integration request and from an integration response to the corresponding method response. A mapping template is a script expressed in Velocity Template Language (VTL) and applied to the payload using JSONPath expressions.
– API Gateway Developer Guide

Show me the code

Here’s what I came up with for converting Slack’s webhook (it’s application/x-www-form-urlencoded, yuck!) to an Elasticsearch query:

#set($httpPost = $input.path('$').split("&"))
{
  "from" : 0, "size" : 10,
  "query": {
    "query_string": {
      "fields": ["attachment.content", "_id"],
  #foreach( $kvPair in $httpPost )
    #set($kvTokenised = $kvPair.split("="))
    #if( $kvTokenised[0] == "text" )
    "query" : "$util.escapeJavaScript($util.urlDecode($kvTokenised[1]).replace('!trigger','').trim())"
    #end
  #end
    }
  },
  "_source": ["_id", "attachment.date", "URL"],
    "highlight" : {
     "fields" : {
         "attachment.content" : {}
     },
     "pre_tags" : ["*"],
     "post_tags" : ["*"]
  }
}

Here’s the conversion of Elasticsearch’s response back to application/json (yes, Slack’s webhook sends post data and expects JSON back; totally weird):

#set($hitcount = $input.json('$.hits.total'))
{
#if($hitcount == "0")
  "text": "Sorry, I couldn't find *anything* about that.",
#else
  #set($hitlist = $input.json('$.hits.hits'))
  "text": "Found $hitcount results:
    #foreach( $hit in $util.parseJson($hitlist) )
    - $util.escapeJavaScript($hit.get('_id'))
    #end",
#end
"mrkdwn": true,
"context" : {
    "account-id" : "$context.identity.accountId",
    "api-id" : "$context.apiId",
    "api-key" : "$context.identity.apiKey",
    "authorizer-principal-id" : "$context.authorizer.principalId",
    "caller" : "$context.identity.caller",
    "cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
    "cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
    "cognito-identity-id" : "$context.identity.cognitoIdentityId",
    "cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
    "http-method" : "$context.httpMethod",
    "stage" : "$context.stage",
    "source-ip" : "$context.identity.sourceIp",
    "user" : "$context.identity.user",
    "user-agent" : "$context.identity.userAgent",
    "user-arn" : "$context.identity.userArn",
    "request-id" : "$context.requestId",
    "resource-id" : "$context.resourceId",
    "resource-path" : "$context.resourcePath"
    }
}

How’d it go?

I’m not particularly proud of some of these hacks, like looping through all keys until we see a particular one or double-parsing JSON in a single response; but, it seems like practically every example uses a loop through all the fields (almost functional style, vs. imperative). Eventually, I ended up at the reference material for Apache Velocity, having to try a few different ways to extra the data I wanted from the Elasticsearch response. I suppose Velocity is a good choice given that AWS probably runs the underlying product using Java, and that Velocity’s stated goal is to be useful for manipulating everything from SQL to Postscript to XML.

Another problem I discovered was that it’s impossible to send any data with the Slack webhook, so there’s no way to pass an API key header for API Gateway. Unfortunately, Slack uses an entirely different authentication mechanism for requests. In addition, AWS Elasticsearch doesn’t have VPC support, so you’re stuck with other authentication mechanisms like signing all requests (having ES on the open internet, checking request signatures doesn’t give me warm fuzzies).

If I had to do this over again, I would probably use Lambda as an intermediary between API Gateway and Elasticsearch. The feature set in API Gateway just isn’t there yet — writing all of my business logic through manipulating strings with a template language just doesn’t feel like a rock solid solution.

Published inTechnology