Trampoline into AWS

Amazon Web Services started up 10 years ago with the Simple Storage Service. To celebrate this QwikLABS’ on-demand computer labs were free for a whole month. At the time of writing there’s less than a week left, but still it might be useful to publish a curated list of labs.

Currently there are 92 labs available. There are several ways to explore them and the most appealing one is to only do the labs in a quest that will help you achieve a certain badge. That’s right, you can show off that you know how to click through a step-by-step online document instruction! You can even share your board of badges. It seems all this really means is that there’s a public URL people can view.

Another approach might be to work through levels. Labs come in three levels that hint and difficulty, time and complexity. In brief they are:

Introductory level – A short introduction into one service. Often it’s clicking through the same steps as the introductory video of a service, but with links and some theory to read through. Most labs at this level are free.
Fundamental level – Usually have the title “Working with …” these labs jump a little higher. Instead of just clicking around in the AWS console, you might have to use command line tools or some API. These will be provided, but you will need to login via SSH or Remote Desktop.
Expert level – Expect a slightly more complex lab with a combination of services. Often what you’ll learn here can be used as a reference point for real-world implementations. Consider this to usually be at the security level of AWS’ Developer guides and not production ready at all.

So those are the two main approaches you might take, quests or working through levels. Personally neither are appealing to me, although I did use quests to try and get lots of badges. That’s why I decided to list a few labs based on categories. I’ve already distributed this selection internally at my place of employment, but it might be useful for more general distribution.

Security

Amazon Web Services offers security and data protection in the cloud and can prove it.

For instance, it complies to the following sample of laws and regulations:

The infrastructure has certain certifications, examples include:

A full list of compliance is available, there’s just too much to list.However, depending on which service you’re using, you are still responsible for compliance and security of your application level, and perhaps even closer to the metal. The next few labs cover what I consider to be relevant topics on access and integration with Amazon Web Services.

Technologies

Amazon offers lots of services. I made a selection of the ones I consider interesting building blocks for cloud-based applications.

Apps

Going through the individual services in labs get you a basic feeling of how they work, but it is more exciting to see them work together. The following labs are examples of combining different services in to functional applications.

DevOps

DevOps, everybody has a definition and if you’re great at passing exams you can even become a AWS Certified DevOps person. The following two labs are very devops-ey and great starting points to explore more.

Aside from these lists, you could also do the reverse. Documentation at AWS often includes links to labs at QwikLABS.

QwikLABS’ Working with Amazon DynamoDB

Until the 31th of March 2016 all labs on QwikLABS are free, a nice opportunity to see how many badges I can get. My first try was the “Working with Amazon DynamoDB” course.

DynamoDB is a NoSQL database service that abstracts most of the setup, maintenance and scaling behind a less complex interface. Although there is a huge amount of documentation, there are few ways to get a good understanding of what DynamoDB can do without starting a project.

The first professional experience I had with DynamoDB turned out to be a false start. The project wanted to use too many hip new technologies, but designed tables the way that is best practice for relational databases. It wasn’t possible to run DynamoDB locally, which is something that can frustrate programmers who are uncomfortable using AWS or simply don’t have enough permissions.

At the same time I wasn’t sure how to configure read and write capacity on tables. These parameters are used by the service to provision and scale the capacity of the underlying technology.

These issues led to a decision that NoSQL wasn’t something to do for that project at that time.

The basics

The lab guides you through the basics of creating and using DynamoDB tables with the AWS console. You create tables through a wizard and learn a little about the configuration options. Then you do some queries and move on to the example.

The example

In this lab you’re using credentials from a Twitter account on an application which streams tweets to a web interface and also tries to store them in several DynamoDB tables. The rate of tweets streaming through is determined by a slider. Sliding right means tweets come in faster. Tweets that failed to be stored in DynamoDB are shown slightly transparent.

To avoid messing with my existing account, I made the effort of creating a new account specifically for this lab. The instructions mentioned at the beginning you might want to set up the account and API access before being with the AWS part, but the instructions on how to do that are very late into the lab.

The example is pretty straight forward. The steps are easy to follow. The majority of the duration should have been in getting all resources set up and waiting for Twitter API access. In my case, I got stuck trying to get the application to work.

After logging in with my Twitter account, the screen stayed blank. No tweets were coming in, but they should start shortly after the authentication with Twitter. It took me several minutes and working through the troubleshooting guide to discover an error in the application itself.

The single page app is backed by Socket.io running on Node. When the user logs in with his Twitter account the API credentials are used to poll for new tweets. Redis is used to keep track of the different users and their web sockets. When a new tweet arrives, the Node app will store it in DynamoDB and also emit the data over the web socket by calling the publish method on the publisher variable with a user id and text version of a JSON object.

For the purpose of the lab there is no need to support multiple users. This meant that hacking the Node app into a one-user solution was enough. All I needed to do is honor the publish contract and emit to the web socket of the last user signing in. In code it looks something like this:

var publisher = {}

socket.on("i am", function ( ... ) {
  publisher.publish = function (userid, message) {
    socket.emit( ... , JSON.parse(message));
  }
}

Once I fixed the Node app I could play around with the slider to see the effect of changing write and read capacity on DynamoDB.

Having a constant stream of data showed me that scaling reads and writes is a process you need to tweak and monitor. Although scaling goes quite fast, at higher speeds many of the write requests couldn’t be fully processed. Depending on the application there can be a big difference in provisioning read and write capacity. In this application the write capacity was crucial, each tweet triggered a write.


This lab helped me understand the basics of creating and querying DynamoDB tables. The example showed the importance and impact of configuring the read and write capacity at table level to match (predicted or measured) requests to it. With capacity too low on writes (or reads) the application won’t be able to handle load properly. On the other hand, provisioning too much capacity might just be a waste of cycles and coins.