GitLab
vs
Buildkite
vs
Decision Kit
Decision Kit
Summary
Buildkite is a continuous integration tool designed to improve software developer productivity. Buildkite product consists of three main components: - Agent: Is a small, cross-platform build runner that is enables an enterprise to run its builds in their own infrastructure. Agents run on several platforms including Ubuntu, Debian, Mac, Windows, Docker, and others. Agents can only run one build job at one time. - Pipelines: Are containers to define and model the build workflows and also to kick off post build steps. - Deployments: These are steps that run after a build is complete and can be either triggered automatically or manually.
As a smaller company with limited resources, Buildkite has taken a strategy of using plug-ins to extend its functionality and to integrate with other software. It is unclear how many community plug-ins have been developed. However as on 3/10/2020 there are about 19 supported plug-ins provided by Buildkite.
Strengths
- Parallel Testing: One of the key value props for Buildkite is the ability to run parallel tests on builds, thus reducing test times from hours to minutes. Buildkite does this using its parallelism approach to spawn out multiple agents (e.g. several hundred parallel agents). With different tests running on a different agents. This massive parallel testing is possible due to the light weight build agent.
- Runs on-premises and on AWS infrastructure
- Agents can be deployed on multiple platforms.
- Pipelines as code: Ability to define complex pipelines in yml including running any scripts, tool or shell commands within the pipeline.
Gaps
- Buildkite is a point solution for continuous integration and currently lacks other critical DevOps steps such as Issue Management, Source Code Management, Security Testing etc. These capabilities require integrations with other tools.
- Security - Buildkite relies on third party providers for security testing. Calling third party code to launch security tests would require special scripts and execution of APIs from within Buildkite’s pipeline definition.
- Lacks native support for an Artifact Repository. File paths to various artifacts have to be noted and maintained by users.
- Secrets management is possible but users have to either write plug-ins to standard secrets management products such as Hashicorp or write custom scripts to pull secrets from a secure storage and apply them at the right step in the build process.
- Collaboration features are limited to developers annotating comments inline within log files and other points of failure.
Resources
Feature Comparison
| FEATURES |
|
|
|---|---|---|
|
Free CI/CD with GitLab hosted or self-managed Runners
GitLab.com hosted runners allow you to use GitLab CI/CD completely free up to 400 build minutes for private projects and 50,000 minutes for public projects. Use your own runner for unlimited build minutes or special build environment requirements. |
|
|
|
Built-in CI/CD
GitLab has built-in Continuous Integration/Continuous Delivery, for free, no need to install it separately. Use it to build, test, and deploy your website (GitLab Pages) or webapp. The job results are displayed on merge requests for easy access. |
|
|
|
CI/CD Horizontal Autoscaling
GitLab CI/CD cloud native architecture can easily scale horizontally by adding new nodes if the workload increases. GitLab Runners can automatically spin up and down new containers to ensure pipelines are processed immediately and minimize costs. |
|
|
|
CI/CD Pipelines Dashboard
Visualize the history and current status of pipelines across projects and groups all in a single dashboard that can be customized for each user. Learn more about Cross-Project Pipelines in the Operations Dashboard |
|
|
|
Online visualization of HTML artifacts
Access your test reports, code quality and coverage information directly from your browser, with no need to download them locally. |
|
|
|
Scheduled triggering of pipelines
You can make your pipelines run on a schedule in a cron-like environment. |
|
|
|
Code Quality MR Widget
Code Quality reports are available in the merge request widget area, giving you early insights into how the change will affect the health of your code before deciding if you want to accept it. |
|
|
|
Code Quality Reports
Full Code Quality reports are available on the pipeline page, showing areas of the codebase that do not meet the organization’s preferred style or standards. |
|
|
|
Code Quality violation notices in MR diffs
Code Quality violations introduced in a merge request are annotated in the merge request diff view to detail how the code quality could decrease if merged. |
|
|
|
Protected variables
You can mark a variable as “protected” to make it available only to jobs running on protected branches, therefore only authorized users can get access to it. |
|
|
|
Customizable path for CI/CD configuration
You can define a custom path into your repository for your CI/CD configuration file. |
|
|
|
Run CI/CD jobs on Windows
GitLab Runner supports Windows and can run jobs natively on this platform. You can automatically build, test, and deploy Windows-based projects by leveraging PowerShell or batch files. |
|
|
|
Run CI/CD jobs on macOS
GitLab Runner supports macOS and can run jobs natively on this platform. You can automatically build, test, and deploy for macOS based projects by leveraging shell scripts and command line tools. |
|
|
|
Run CI/CD jobs on Linux ARM
GitLab Runner supports Linux operating systems on ARM architectures and can run jobs natively on this platform. You can automatically build, test, and deploy for Linux ARM based projects by leveraging shell scripts and command line tools. |
|
|
|
Run CI/CD jobs on FreeBSD
GitLab Runner supports FreeBSD and can run jobs natively on this platform. You can automatically build, test, and deploy for FreeBSD-based projects by leveraging shell scripts and command line tools. |
|
|
|
Show code coverage rate for your pipelines
GitLab is able to parse job output logs and search, via a customizable regex, any information created by tools like SimpleCov to get code coverage. Data is automatically available in the UI and also as a badge you can embed in any HTML page or publish using GitLab Pages. Learn how to generate and show code coverage information in GitLab |
|
|
|
Details on duration for each command execution in GitLab CI/CD
Other CI systems show execution time for each single command run in CI jobs, not just the overall time. We’re reconsidering how job output logs are managed in order to add this feature as well. |
|
|
|
Protected Runners
Protected Runners allow you to protect your sensitive information, for example deployment credentials, by allowing only jobs running on protected branches to access them. |
|
|
|
Built-in and custom project templates
When creating a new project, you can choose to kickstart your project from a predefined template that already has some working example code and CI preconfigured. In addition, you can define a custom project templates by assigning a group. Child projects of this group are available as templates when creating a new project. |
|
|
|
Automatic Retry for Failed CI Jobs
You can specify a retry keyword in your .gitlab-ci.yml file to make GitLab CI/CD retry a job for a specific number of times before marking it as failed. |
|
|
|
Pipelines security
The ability of running CI/CD pipelines on protected branches is checked against a set of security rules that defines if you’re allowed or not. It includes creating new pipelines, retrying jobs, and perform manual actions. |
|
|
|
Include external files in CI/CD pipeline definition
You can include external files in your pipeline definition file, using them as templates to reuse snippets for common jobs. |
|
|
|
Browser Performance Testing
Easily detect performance regressions for web apps and pages prior to merging into master. Browser Performance Testing is included in Auto DevOps, providing automatic performance analytics of the root page with zero configuration. |
|
|
|
Load Performance Testing
Easily detect performance regressions for APIs prior to merging into master. |
|
|
|
Step folding for CI/CD logs
Collapse the job log output for each command. |
|
|
|
CI/CD for external repo
Connect your projects hosted on external services (like GitHub or Bitbucket) and leverage the power of GitLab CI/CD pipelines to build, test, and deploy your applications easily. |
|
|
|
CI/CD for GitHub
Connect your projects hosted on GitHub and leverage the power of GitLab CI/CD pipelines to build, test, and deploy your applications easily. |
|
|
|
Bad test quarantine
Don’t let red builds become the norm. Across all tests, keep flaky or broken tests out of sight (but not out of mind), and keep the build green with one-click quarantine of tests. |
|
|
|
Unit Test Report
GitLab allows you to view unit test results for a pipeline, giving you insight into the test execution for the pipeline. |
|
|
|
See unit test summaries in merge request widget
GitLab allows you to view unit test results from the merge request widget, giving you insight into quality impacts of your changes. Learn more about unit test summaries in the merge request widget |
|
|
|
Interactive Web Terminals
Interactive web terminals allow you to connect to a running or completed Kubernetes, Docker, or Shell runner job and manually run commands to better understand what’s happening in the system. |
|
|
|
Run jobs only when there are changes to a file or path
Jobs can be configured to run only when there are changes to a specific file or path, giving you control over execution to allow for more complex build pipelines optimized for the changes in each commit. |
|
|
|
Windows Container Executor
With this feature you are able to use Docker containers on Windows directly, in much the same was as if they were on Linux hosts. This enables more advanced kinds of pipeline orchestration and management for users of Microsoft platforms. |
|
|
|
Pipeline deletion
Deleting a pipeline is possible using the API and also in the UI on the Pipeline Details page. This allows for cases where perhaps secrets have been leaked in a pipeline, many unneeded pipelines have been created, or other issues have occurred where pipelines need to be deleted. |
|
|
|
Explicit support for monorepos
The ability to execute jobs only/except when there are changes for a given path or file support monorepos where many microservices are contained in a single repo. |
|
|
|
First class container building
The ability to specify that a container should be built during a CI/CD job without needing to specify the implementation details. |
|
|
|
Trigger pipeline on any event in code repository
Enables pipelines/workflows to be started based on when any defined event is executed in the code repository. For example, could run a workflow to send a welcome email on adding a new member to a repository or project. |
Through WebHooks |
Supports 21 event triggers |
|
Trigger pipeline on any event in code repository app eco-system
Enables pipelines/workflows to be started based on when any defined event is executed in the code repository or in any app extension of that repository’s eco-system. For example, when an event happens in the Slack integration, update a repo work item. |
Through WebHooks |
|
|
Community powered workflows (configuration is code so are shareable)
GitLab pipeline (workflows) are defined as yml in repos and can be shared just like actions. |
|
|
|
Any platform, any language, and cloud
Can run on any OS platform, for any language, and on any cloud provider |
|
|
|
Auto suggest pipelines to start with based on code language
Through language detection, auto suggest pipeline templates to run to help users quickly get a pipeline running. |
|
Auto Devops runs a pipeline, not just suggest one |
|
Advanced CI/CD configuration linter
The CI linter provide warnings and error messages when validating your |
|
|
|
Comes with many pre-defined pipelines
Offers many pre-defined pipelines that capture best practice and make it easy for a user to get started with each project for common languages, platforms, and configurations. |
|
|
|
Connects the diff tools & services used during the SDLC
Can be used as a central glue to orchestrate, and connect data and outputs from your many different tools & services. |
|
|
|
Live streaming of logs from running pipeline
Ability to see live job logs (while the pipeline is running). |
|
|
|
Search across all job logs
Search across all or more than one job log at once. Enables more efficient search for errors and other content of interest while troubleshooting or reviewing job output. |
|
browser search within the log at a time |
|
View raw logs in plaintext
Ability to get the plain text of a log, no mark up, to be able to share it or use it externally. |
|
|
|
Multiple pipelines per repo
Ability to define multiple pipelines per code repository to enable either different processes to be run at different times, and/or to enable monorepos where there are multiple applications within one repo which need to be built and handled differently per application. |
|
|
|
Reference actions/jobs in another repo
Ability to have pipelines/workflows reference and use actions/jobs from a repo different from the one it is being run from, without needing any installation. |
|
|
Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license
