Transforming CRM DevOps at Airbnb: A Powerful Framework for Continuous Delivery
How we championed the CRM CI/CD framework integrating Salesforce DX, GIT, BUILDKITE and Vlocity for an enhanced, efficient and continuous delivery with high software quality.
By: Sharda Kumari Pramod Gavade
Introduction
The CRM platform offers a robust suite of functionalities for building scalable applications with minimal reliance on complex coding. However, managing and deploying code and configurations within this ecosystem can be challenging, and the constantly evolving nature of the platform presents an extra layer of complexity. This can lead to slow deployment times, difficulty in balancing code and configuration (e.g. Apex classes and triggers vs. validation rules, page layouts), and managing multiple environments, among other issues.
To address these challenges, at Airbnb we have developed a resilient DevOps framework specifically tailored to the CRM platform. The framework automates the process of moving code and configuration into production for developers, system administrators in addition to low-code users such as business analysts building dashboards.
The Challenges
The CRM Platform is a versatile ecosystem with powerful functionalities, but managing code and configurations within it can be challenging. With complex metadata and multiple environments, deploying changes efficiently is difficult. Additionally, the evolving nature of the platform requires a proactive approach.
Deployment times: Deploying changes can be a time-consuming process, especially for extensive applications. This can impact the speed of delivery causing dissatisfaction among developers and stakeholders alike.
Balancing code and configuration: The CRM platform permits the development of applications via both programmatic code (using Apex, Visualforce and Lightning Web Components) and low-code development methodologies (using tools like App Builder and Flow Builder). However, effectively managing a combination of code and configuration based development poses a formidable challenge.
Managing multiple environments: CRM developers typically engage in individual environments (sandbox or scratch orgs) to build and test their code prior to moving it to higher-level environments. However, administering multiple environments across different teams can become increasingly intricate and time-consuming.
Complexity of file metadata: CRM is an intricate platform with different types of metadata (including, but not limited to, Apex classes, triggers, Lightning components, flows) that necessitate meticulous management during the development and deployment process.
Keeping up with changes: As a cloud-based platform, CRM undergoes frequent changes, with new features and updates being released regularly. Keeping up with these changes and ensuring their non-disruptive integration with existing applications can be a significant challenge.
The Solution
At Airbnb, we have developed a resilient DevOps framework tailored specifically to the CRM platform that integrates Salesforce DX, Git, and Buildkite. Our approach facilitates the inclusion of all stakeholders, including developers to system administrators and low-code users, into the development and deployment process, thereby optimizing the DevOps solution for all personas involved.
The CRM DevOps lifecycle comprises the following critical environments that are necessary for the effective deployment of code:
- Developer(DEV) — The Developers utilize the DEV environment to build and refine the code.
- Integration(SIT) — SIT ensures seamless integration with other systems.
- Quality Assurance(QA) — QA verifies the software’s functionality.
- Full copy(STAGING) — Staging provides a realistic setting for training and user acceptance.
- Pre-release — Pre-release serves as a controlled hosting platform before the code goes live.
- Hotfix — Hotfix enables swift resolution of urgent production issues.
- Prod — Production instance that houses all live traffic and data.
Each of these environments is associated with a corresponding branch in the Git version control system and is seamlessly connected via a Buildkite DevOps pipeline. After successfully passing through the peer review process by developers, Buildkite jobs are triggered which utilize Salesforce DX to deploy the code into the target sandbox.
The DevOps journey kicks off with the refreshing of individual developer instances within the CRM from the integration (SIT) environment to import changes made by other developers. Following this, a feature branch is created in Git from the integration branch, facilitating development and unit testing in individual environments. Developers can work efficiently in their own development environments while ensuring seamless integration and collaboration across the board.
Once new functionality has been implemented, the development phase is complete. Code undergoes a series of rigorous quality assurance procedures before moving on to the SIT environment. One of the procedures employed is static code analysis, which ensures adherence to coding standards and best practices. Additionally, pull requests are subjected to extensive review and approval procedures to maintain code quality. Before being promoted, the code is finally deployed to the SIT environment for integration testing, where system integrations are validated.
After successful testing, changes are promoted to the QA environment for functional and regression testing, which includes automation testing using Provar. QA environment is used for testing a feature end to end between the integrating systems. Automation testing scripts are run in this environment to ensure a bug free product. While code reviews make sure high coding standards are in place, QA and Automation testing ensures a well functioning solution.
The next step in the DevOps journey is the staging environment, where user acceptance testing and performance testing take place. PTPaaS sandbox is used for performance and load testing, which is a pilot offering specifically designed for performance testing. This sandbox is connected to the staging branch to ensure seamless execution of performance testing. With the final steps of validation complete, changes that originated as changes in someone’s feature branch can now be promoted to production and used by everyone.
In contrast to normal feature development, we need to be able to push urgent fixes. This flow takes a different route. Fixes originates in an environment dedicated for Hotfixes which is a replica of Prod (called Hotfix). A fix is worked on in the hotfix environment, tested and validated by QA and then pushed to Prod via Staging. Hotfixes circumvent the SIT environment. Once promoted to production, fixes are then back propagated to the lowest developer sandbox in the chain.
A Release time block (Deployment window) is determined for the code push and is agreed upon by all stakeholders for the code push. Having a dedicated time for deployment helps with setting expectations with the end users on the new features being released. One of the most significant challenges of CRM deployments is prolonged deployment times, which is further compounded with sizable implementations and longer test executions. Changes can sometimes take upto 90 min to be deployed. To circumvent this challenge, our framework takes a proactive approach by performing a build validation against the Production environment. This validation checks all test class executions and metadata validation in advance, typically more than 24 hours before the deployment window, and enables Quick Deploy in the target Org. During the deployment window, our DevOps automation executes a quick deploy from Buildkite, which helps to detect issues with the build or test runs early on and significantly reduces the actual deployment time. To further minimize deployment times, we have implemented Incremental Deploys, which deploy only the differences between codebases instead of the entire codebase. The previous commit ID is stored in a custom setting in the target org, and Buildkite retrieves it during deployment to commit only the hash set.
The Wins
- Using Quick Deploy reduced the deployment time from an average of 90 mins to 15 mins
- Enabled delta deploys instead of full package.
- Config and Metadata versioning.
- Built in static code analysis in the repository.
- Rollback mechanism.
- Integration test hooks.
Conclusion
Our DevOps implementation has successfully enabled efficient, continuous delivery of high-quality software on the CRM platform. This was achieved by seamlessly integrating Salesforce DX, Git, and Buildkite, thereby establishing a DevOps framework that is optimized for all personas, including developers, admins, and low-code users. As a result, we have witnessed a remarkable decrease in deployment time and a noteworthy improvement in software quality, ultimately leading to the delivery of greater value to our clients.
If this type of work interests you, check out some of our related positions.
Acknowledgments
Special mention to Yiming Yan, Sameer Miraj, Vasu Rampally,Srinath Therampattil, Leah Kennedy and the entire team for helping onboard onto this framework. Special thanks to Sudheer Peddineni for the guidance.
