The eight principles of Sustainable Software Engineering

The beginning of the story

My team have been working with Streaming Applications that are running partly on Azure Cloud services, consuming millions of transactions daily to provide enriched information to our end users. The complexity of such a system makes it difficult to discover the sustainability impact and improve where is needed. Additionally, we had to find the right support from the management and the business. Therefore, we started with convincing our Product Owner by presenting the benefits of sustainable development through potential ideas and real implementation plans.

Luckily, on Tribe level they initiated an “Innovation Sprint” where teams were invited to explore innovative solutions, ideas. We used this Innovation Sprint to present our use case and how we made a difference with applying the sustainability mindset. At the same time, we found an Area Lead who, with the support of our use cases, managed to bring a real change to the engineering culture by introducing sustainability as a tribe goal. You can find her article h, that presents the story from the area lead point of view.

Before moving to the details of our implementations, I would like to present the eight principles of sustainable engineering according to Microsoft. This concept was important for our team since we used these to analyse our applications.

1. Carbon: Build applications that are carbon efficient.
2. Electricity: Build applications that are energy-efficient.
3. Carbon Intensity: Consume electricity with the lowest carbon intensity.
4. Embodied Carbon: Build applications that are hardware efficient.
5. Energy Proportionality: Maximize the energy efficiency of hardware.
6. Networking: Reduce the amount of data and distance it must travel across the network.
7. Demand Shaping: Build carbon-aware applications.
8. Optimization: Focus on step-by-step optimizations that increase the overall carbon efficiency.

You can find the detailed explanation of each of these principles here.

So now let’s see how we applied these principles in actual use cases.

Network efficiency and how did we tackle it?

According to the Microsoft guidelines and principles, “carbon-efficient applications focus on reducing the amount of data and distance it travels” (Principle 6: Network efficiency - training. Training | Microsoft Learn. (n.d.). Retrieved January 24, 2023)

Thus, we wanted to stop the redundant data offloads on our own system. We knew that some data that we offload is also available on our Global Data Platform (GDP) – a centralised data platform. (During the last years, Rabobank has been working hard to set up a centralised data portal where you can shop for different datasets.)

The decommissioning of this redundant data offloads was a logical step to focus on. This task was long on our backlog, but we never managed to get priority for it. You might recognise this in your own project. When something seems not to add immediate business value why would you do it? However, using the sustainability movement within our team we could easily convince our Product Owner. Thus we managed to get the right priority. In the end, this eliminated in our system a Databricks cluster, multiple Azure Event Hubs, Azure Storage Account, Virtual Networks and Security Setups such as firewall rule, and maintenance of a Service Principal. Apparently, all these technologies were needed to keep all these redundant datasets.

Needless to say, this step improved our solution not only by reducing the costs of the resources but also creating a more sustainable and maintainable architecture.

Carbon: the story of the Green Watchdog

Carbon emission is a by-product of any activity. According to Microsoft the goal of this principle is to reduce carbon emission while maximise the value of the deliverable (Principle 1: Carbon - training. Training | Microsoft Learn. (n.d.). Retrieved January 24, 2023). This concept helped us to focus on a specific problem that we faced since we started working on Azure infrastructure. As I mentioned earlier, one of our applications is a streaming solution using a Kafka messaging implementation from where is continuously polling the transaction data. This data then is enriched by a Spark framework, using Spark Structured Streaming. In any development process once we reach the level of system testing, we deploy our applications to a test environment and we run automated integration, and performance tests.

You can imagine a streaming application running always on a development environment, can lead to a huge carbon emission that is providing no value or benefits. First, we were trying to manually shut down these resources, it goes without saying this was not always a big success.

Here is a side note: Databricks clusters are configurable to have Automatic termination option, but this will not be applicable for a streaming application with a continuous poll. When we applied our new “magnifying glass” of sustainability, we could quickly identify the issue, and came up with a simple solution: The Green Watchdog.

What is The Green Watchdog?

The Green Watchdog is an application that is running in the DevOps Agents connecting to our Azure Dev subscriptions, and screening for running clusters to shut (or bark) them down. The application is triggered every evening on all our development environment. With implementing a few lines of code (written in Python, using Databricks CLI) we reduced almost to zero our costs on development environments, leading to less carbon emission.

The Demand Shaping and how we applied it: The Green AutoScaler

Finally, I would like to introduce again a very specific issue that is applicable on streaming solutions. Building streaming applications that are using the transaction data, has different demand/supply needs during a course of a day. Meaning that in the morning hours the number of transactions is growing and during the night they drop. By applying the demand shaping sustainable engineering principle, we saw that the configuration of our cluster is not efficient. We have been using the fixed configuration for the number of clusters, thus regardless the demand, we encountered situations when we were having huge lags (up to 2 hours), or we saw the load of the cluster dropped up to 90% leaving idle machines running for hours. At this point we tried the Optimized Autoscaling option, which was overly aggressive for our 24–7 streaming workloads. Therefore, we came up with the self-autoscaler (The green autoscaler) to monitor the load of the cluster and adjust the size of it accordingly.

This application is currently running every hour and based on the actual lag and load adjusts the size of the cluster, by downscaling or upscaling it. By applying these principles, we estimated that we reduced our costs total 70% and therewith we reduced carbon emission dramatically too. 

So now, let’s move to the next sections, what can we learn from all of these? Our use cases were specific, however there are some generic lessons learnt I would like to share that can help anyone to apply these concepts.

Sustainability and the engineering culture – What can you learn from it as a developer?

Keep it Simple

Once we started to analyse our projects based on the principles of Sustainable Software Engineering, we realised that the concept is relatively simple and can easily help us to improve the applications that we are delivering. Just simply by looking at our components and checking the purpose of each item we were able to eliminate storage, reduce network traffic, and we adjusted the usage of the resources based on demand, thus we turned off idle machines in the system. All these improvements are straight forward, the only thing what was required from us to have the right attention, focus and priority.

The right priority

Obviously, the support from the area lead and from our Product Owner was crucial; and by finding the right channels to share our ideas with them we managed to dedicate time and commitment. It was a great experience to also find an area lead who shared our dedication, and who helped us to get the right attention not only within the context of our team but also on a wider organisational perspective. As mentioned in her blog, by sharing your passion within the organisation you might find others with whom you can learn and grow together.

Dedication is key

We were fully committed and devoted, therefore we managed to allocate time and resources. Our final goal was to reach a structural change in our way of working so that in the future projects we could proactively focus on Sustainable Software Engineering principles. As Microsoft stated, “Sustainability is enough, all by itself, to justify our work” (Overview of Sustainable Software Engineering - training. Training | Microsoft Learn. Retrieved January 24, 2023, from)

We also have seen that sustainable development comes with loads of added benefits. As you learnt, we reduced the complexity and improved the maintainability of the infrastructure by decommissioning unnecessary resources, made the system more resilient by applying the demand-shifting workloads. Although, for our team the main motivation for sustainable engineering is simply sustainability.