The Art of Automating Automation
Introduction
As part of the PayPal Site Reliability & Cloud Engineering (SRCE) team, we are always striving to create new efficiencies through automation. However, writing software that automates what humans do intuitively is not a trivial task. Any component that automates any operation in PayPal infrastructure must provide the highest level of reliability, security, efficiency, and self-service capabilities. In this article, we are going to talk about how we achieve automation within Global Network Services (GNS).
The GNS team has a huge volume of operations that we need to automate. These operations range from mundane tasks that require a few hours of human effort, to complex tasks that require a multi-day effort from the operations team. The biggest challenge for us is, how do we automate big and small operations without consuming vast amounts of development cycles. This is an especially important aspect as we always strive to justify the “ROI (return on investment)”.
Software Service Requirements
Morphing any network function into a service involves the following list of tasks that need to be completed for producing a working service:
User interface development
- All the services that are exposed to the PayPal development community must have some form of user interface (UI) to let the users consume the network like any other service.
Database interaction and table management
- The majority of services require database table design and writing the code for interacting with Database to manipulate data.
Core business logic
- The actual business logic of a specific service needs to be written and tested.
Integrate with automation tooling Terraform
- Our user base includes a Network operations team which will not consume the services from UI. Instead, they would operate in git-ops model. This means all the services we develop need to integrate with Terraform by developing Terraform providers.
Authentication and authorization
- This involves integration with standard corporation authentication like Single-Sign-on, Multi-factor authentication, Role management, etc.
Service accounts, throttling and rate limiting for API consumers
- We decided to provide APIs for everything. So, all the services that we develop have to provide service accounts and ensure we can rate-limit and throttle the APIs.
As you can see from the above list, producing a viable network service requires us to spend a huge amount of development cycles, in the essential but peripheral software components. In an ideal situation, we would just spend our cycles on writing core business logic and nothing else. We implemented two-pronged approach to reach our ideal state of writing less code.
The first method in solving this problem involves migrating to true micro-service components and pluggable library architecture. The second method we implemented is pure template-based automatic code generation. This method involves generating UI, backend, and DB layers code, based on common templates.
Code Generation
Data modeling has been a pillar of a well-designed software product for a very long time. Once the basic data building blocks are identified, it becomes easier to build the software components that work around the data. If the data is captured in a standard format, then there is a huge scope to automate a variety of software components around the data.
This is the exact approach we adopted. All projects for our team start with identifying the data model. The data model is defined in a YAML format so that our dynamic code generation modules can generate code for various parts of the system. The diagram below captures this process of producing services:
UI Code generation
React framework has been widely adopted by the industry, including PayPal. PayPal application teams have a very modular react repository that is built for building customer-facing applications. We took that library as a base and started using it wherever appropriate.
Most system programmers are not familiar with UI programming. To facilitate rapid UI component development, we lean on model-driven design. With model-driven design, a typical GNS developer specifies the UI layout only. The developer does not write any UI code.
The GNS code generation engine takes the user-defined model and, using the React framework, it produces an independent container of React code with a fully functional UI. This approach allows our developers to focus on the business logic of the application.
Backend Code Generation
Developing a micro-service that provides REST API functionality used to be very tedious. A developer used to write software for each endpoint and work as a web service gateway. However, the Flask framework has gained wide acceptance from Python developers in terms of working as a standard framework. Similarly, Golang has gorilla/mux framework in addition to built-in Golang support.
Most of GNS services are written in Golang or Python. With the help of the above-mentioned frameworks, the GNS code generation engine has been designed to produce common code using Jinja templates.
With the data model already specified in YAML format, the code generation engine uses pre-defined Jinja templates to produce a working stub of a service. GNS developers can take the stub implementations and override them with exact business logic. This reduces programming errors and eliminates the tedious work of writing everything from scratch.
DB Layer Code Generation
Most GNS services require some form of a database to store the information that is depicted during the modeling stage. Writing code to add, delete and modify entries from the database can be tedious. Handwritten code is often susceptible to bugs, style variations, etc. Python and Golang have open-source libraries that deal with Object Relationship Mapper (ORM). We chose to use sqlalchemy to normalize DB operations.
Since the model is defined in YAML format, our code generation engine produces DB handling methods in addition to the backend code.
Terraform Provider Code Generation
Terraform has become the most used tool to automate infrastructure. As GNS service providers, we have decided that all the automation services we provide will have interfaces for Terraform. This will enable us to embrace continuous deployment pipelines that the organization is adapting.
Terraform architecture is flexible and extensible with the ability to add providers for any automation that we like. See Terraform Documentation for more details.
Since Terraform provider is a well structured library, our code generation engine produces golang based providers generated using the Jinja templates. This code then gets build into a pluggable module that can be installed on a running terraform instance. Once this is done the users can exercise standard terraform operations on GNS services just like any other service.
Conclusion
Majority of tasks that are required to automate any network service have been generated by our code generation engine. This is allowing us to focus solely on working on the core logic of a service. Moving to micro services architecture along with embracing template based code generation and model driven architecture is allowing us to produce services at a faster pace. As of this writing, we can produce a working skeleton of any service under a day. This allows our team to focus on core logic and produce more services in less time.
