Mono Repos in AWS with CodePipeline

  • This is more efficient for us in some cases and is not intended to add any fuel to the mono repo vs. single repo debate
  • This lets us group small and highly related but independently shipped things such as AWS lambda functions without creating thousands of repos
  • You can have multiple CodePipelines on a single repo, but they are all triggered by any commit to the repo using GitHub webhooks.
  • We found few available tools to solve this problem, and eventually decided to solve it ourselves.

Our Solution

Solution Overview
  • An AWS API endpoint backed by an AWS Lambda function to receive and authenticate GitHub webhook push events
  • An AWS S3 bucket to store configuration
  • An AWS Lambda to evaluate incoming GitHub push events and start CodePipeline execution on a match
  • An AWS CloudFormation Lambda backed resource to store configuration in S3

Getting Notified: API Gateway GitHub Webhooks

We built a single API Gateway endpoint backed by a AWS Lambda function with the responsibility of authenticating and validating incoming requests from GitHub. The function confirms Authentication via HMAC security and if valid asynchronously calls the evaluation lambda function. We configured GitHub to have a organizational webhook for push events to the endpoint.

What to do? : Lambda Evaluation Function

This AWS lambda function’s responsibility is to process GitHub event data and execute an AWS CodePipeline on a match. The function uses the GitHub repository and branch contained in the event to find configuration information for a pipeline in AWS S3. If a configuration match is found, the JSON configuration is read, and the match regular expressions are run against all the paths in the webhook event. On a match the configured CodePipeline is executed.

How to configure? : CloudFormation Lambda- Backed Resource

This AWS lambda function’s responsibility is to back the CloudFormation in a CodePipeline definition storing, updating, and deleting the needed configuration in S3 for the evaluation function when the CloudFormation is executed. The YAML for a project to participate in a mono repo and store configuration in S3 would look like:

Conclusion

While you can debate the mono repo pattern, for us having the ability to group highly cohesive projects together in a single repo with separate build and deployments makes our teams more effective. Leveraging the tools available within the AWS ecosystem, we built a solution that allows our SRE and Engineering teams to be more productive.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Todd Palmer

Todd Palmer

Husband, Father, Software Developer, Cyclist, White Gold Wielder