Beanstalk Seeder

Elastic Beanstalk is a great platform, it offers both a Web tier and a Worker tier. I recently wrote about Simple Routing one of my library that allows you to route a SQS message to a specific endpoint on the Worker.

While Beanstalk works great once it’s deployed to AWS there is no easy way to run it locally. As soon as you want to execute an end-to-end flow involving both the Web and the Worker you need to manually POST requests to the Worker using Postman which is cumbersome and error-prone.

As it core all the SQS daemon does is dequeue messages from a SQS queue and POST it to a specified endpoint. With this goal in mind I wrote Beanstalk Seeder.

I had the following objectives:

• Users should be able to get up and running quickly
• Meaningful logging
• Transform the SQS message attributes into HTTP headers (in order to support Simple Routing)

Get up and running quickly

You can get Beanstalk Seeder from GitHub releases. Download the archive and extract it somewhere.

Beanstalk Seeder’s configuration is detailed in the README. All you need is a iAM user, the Worker URI and the SQS queue URI.

Meaningful logging

When running a third-party tool, it’s critical to get meaningful logging as the binary is a black-box for the end user. I use structured logging in order to make querying the log events a breeze. My logging framework of choice is Serilog.

var loggerConfiguration = new LoggerConfiguration()
    .Destructure.ByTransforming<MessageAttributeValue>(Destructure)
    .Destructure.ByTransforming<Message>(Destructure)
    .MinimumLevel.Is(serilogLevel)
    .Enrich.WithDemystifiedStackTraces()
    .Enrich.FromLogContext()
    .WriteTo.Console(serilogLevel);

The previous snippet highlights only a few of the Serilog features.

The structure-capturing operator

MessageAttributeValue and Message are both defined in the awssdk.sqs NuGet package. I’m interested in logging only some of their properties, Serilog has the ability to capture object via the structure-capturing operator.

Enrichment

Enrichment is the act of adding additional properties to events, other than the ones originating from the message template.

Serilog supports ambient context. I’m also using the excellent Ben.Demystifier for getting nicer stack traces.

Sinks

By default, Serilog does not log anywhere. In order to record events you’ll need to configure one or more Sinks. In this case I’m writing to the console but they are many other Sinks available.

Result

I hope the recorded events are descriptive enough so that an end user know what’s happening:

• First I display the settings used, this is important as they could come from the appsettings.json, environment variables or even the user secrets if the environment is Development.
• Then using the structure-capturing operator I log the relevant SQS message properties.
• Instead of logging the complete HTTP request I log the content of the body and the relevant headers.
• When deleting the message, I log the ReceiptHandle, this is the value used to delete a message and the user can correlate it to what was displayed above.
• Finally, rather than not displaying anything when there are no messages in the queue I inform the user that’s the case and how long I’ll wait before retrying.

Interestings bits

I’m using a CancellationTokenSource so that the user can stop the message pump at any time (relying on Console.CancelKeyPress).

The MessagePump class is the only class with some logic. I wrote some tests around the cancellation token, the transformation of SQS message attributes into HTTP headers and the back-off when no messages are available in the queue.

Conclusion

I hope you’ll find Beanstalk Seeder as useful as I did, combined with Simple Routing it simplified and streamlined my Elastic Beanstalk development.

I also wanted to point out that Beanstalk Seeder is platform agnostic. It doesn’t matter if you’re developing using Node.js, Go or any other of the Elastic Beanstalk supported platforms, all you need to do is install the latest .NET Core runtime (available on Windows, macOS and Linux).