In this article I want to describe how Amazon Web Services helped me to automate my tasks.
I have an Android application which provides parking information for the users. A scheduled Android job checks for updates every 4 hours. Updates can be found in a static Json file hosted on Firebase Hosting. This Json file has a specific structure, but it differs from the source, where all data comes from. There is another file called config.json which contains the version number of the data. Although it's a cost-effective way to store minimum amount of data, but it was a pain in the ass to maintain it.
Earlier I had to:
- check manually if there's a data update
- download the data
- transform the data to my Json structure
- put it into a zip file
- update the config.json file with the new version number
- upload them to Firebase Hosting
And what is the problem with it?
- I have to spend time to check the api every time
- there's no automation
- there're no logs or summaries because everything processed manually
(...)
Don't say anything... It's a mess :) This why I decided to move on to Amazon Web Services.
First I had to design what and how I wanted to develop. I don't want to use an entire EC2 instance, Kubernetes, Docker, any kind of virtualization or database. It must run only at specific times.
My final tech stack is:
- Lambda for the smaller functions act as microservices combined with environment properties to make them configurable,
- S3 to store input and generated files,
- EventBridge to trigger the checker service as a CRON job,
- SQS to communicate between the services,
- SNS to trigger the email notification,
- CloudWatch for logs and alerts,
- SSM (Simple Systems Manager) Parameter Store to store dynamic parameters about the process results
And the entire process visualized:
And for administration tasks:
- Budgets,
- Cost Explorer,
- IAM
And lets talk about the software architecture. I've created 4 Lambda functions:
- checkData: This is triggered by EventBridge once a day and checks for new data. If no new data found, the notifyData function sends me an email about there's no update, otherwise the processData will be called.
- processData: This function processes the downloaded input file and creates the output files and calls notifyData and deployData functions.
- notifyData: This function calls SNS to send a notification email about the result.
- deployData: This function deploys the generated files to Firebase Hosting.
And that's all! :) If you have similar tasks, which can be automated, don't hesitate, move it to the cloud!
