Continuous Integration

Extension integration tests can be run on CI services. The vscode-test library helps you set up extension tests on CI providers and contains a sample extension setup on Azure Pipelines. You can check out the build pipeline or jump directly to the azure-pipelines.yml file.

Automated publishing

You can also configure the CI to publish a new version of the extension automatically.

The publish command is similar to publishing from a local environment using vsce, but you must somehow provide the Personal Access Token (PAT) in a secure way. By storing the PAT as a VSCE_PAT secret variable, vsce will be able to use it. Secret variables are never exposed, so they are safe to use in a CI pipeline.

Azure Pipelines

Azure Pipelines

Azure Pipelines is great for running VS Code extension tests as it supports running the tests on Windows, macOS, and Linux. For Open Source projects, you get unlimited minutes and 10 free parallel jobs. This section explains how to set up an Azure Pipelines for running your extension tests.

First, create a free account on Azure DevOps and create an Azure DevOps project for your extension.

Then, add the following azure-pipelines.yml file to the root of your extension’s repository. Other than the xvfb setup script for Linux that is necessary to run VS Code in headless Linux CI machines, the definition is straight-forward:

  1. trigger:
  2.   branches:
  3.     include:
  4.     - master
  5.   tags:
  6.     include:
  7.     - v*
  9. strategy:
  10.   matrix:
  11.     linux:
  12.       imageName: 'ubuntu-16.04'
  13.     mac:
  14.       imageName: 'macos-10.13'
  15.     windows:
  16.       imageName: 'vs2017-win2016'
  18. pool:
  19.   vmImage: $(imageName)
  21. steps:
  23. - task: NodeTool@0
  24.   inputs:
  25.     versionSpec: '10.x'
  26.   displayName: 'Install Node.js'
  28. - bash: |
  29.     /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
  30.     echo ">>> Started xvfb"
  31.   displayName: Start xvfb
  32.   condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'))
  34. - bash: |
  35.     echo ">>> Compile vscode-test"
  36.     yarn && yarn compile
  37.     echo ">>> Compiled vscode-test"
  38.     cd sample
  39.     echo ">>> Run sample integration test"
  40.     yarn && yarn compile && yarn test
  41.   displayName: Run Tests
  42.   env:
  43.     DISPLAY: ':99.0'

Finally, create a new pipeline in your DevOps project and point it to the azure-pipelines.yml file. Trigger a build and voilà:

pipelines

You can enable the build to run continuously when pushing to a branch and even on pull requests. See Build pipeline triggers to learn more.

Azure Pipelines automated publishing

  1. Set up VSCE_PAT as a secret variable using the Azure DevOps secrets instructions.
  2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
  3. Declare a deploy script in package.json without the PAT (by default, vsce will use the VSCE_PAT environment variable as the Personal Access Token).
  1. "scripts": {
  2.   "deploy": "vsce publish --yarn"
  3. }
  1. Configure the CI so the build will also run when tags are created:
  1. trigger:
  2.   branches:
  3.     include:
  4.     - master
  5.   tags:
  6.     include:
  7.     - refs/tags/v*
  1. Add a publish step in azure-pipelines.yml that calls yarn deploy with the secret variable.
  1. - bash: |
  2.     echo ">>> Publish"
  3.     yarn deploy
  4.   displayName: Publish
  5.   condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/tags/'), eq(variables['Agent.OS'], 'Linux'))
  6.   env:
  7.     VSCE_PAT: $(VSCE_PAT)

The condition property tells the CI to run the publish step only in certain cases.

In our example, the condition has three checks:

  • succeeded() - Publish only if the tests pass.
  • startsWith(variables['Build.SourceBranch'], 'refs/tags/') - Publish only if a tagged (release) build.
  • eq(variables['Agent.OS'], 'Linux') - Include if your build runs on multiple agents (Windows, Linux, etc.). If not, remove that part of the condition.

Since VSCE_PAT is a secret variable, it is not immediately usable as an environment variable. Thus, we need to explicitly map the environment variable VSCE_PAT to the secret variable.

GitHub Actions

You can also configure GitHub Actions to run your extension CI. In headless Linux CI machines xvfb is required to run VS Code, so if Linux is the current OS run the tests in an Xvfb enabled environment:

  1. on:
  2.   push:
  3.     branches:
  4.       - master
  6. jobs:
  7.   build:
  8.     strategy:
  9.       matrix:
  10.         os: [macos-latest, ubuntu-latest, windows-latest]
  11.     runs-on: ${{ matrix.os }}
  12.     steps:
  13.     - name: Checkout
  14.       uses: actions/checkout@v2
  15.     - name: Install Node.js
  16.       uses: actions/setup-node@v1
  17.       with:
  18.         node-version: 10.x
  19.     - run: npm install
  20.     - run: xvfb-run -a npm test
  21.       if: runner.os == 'Linux'
  22.     - run: npm test
  23.       if: runner.os != 'Linux'

GitHub Actions automated publishing

  1. Set up VSCE_PAT as an encrypted secret using the GitHub Actions secrets instructions.
  2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
  3. Declare a deploy script in package.json without the PAT.
  1. "scripts": {
  2.   "deploy": "vsce publish --yarn"
  3. }
  1. Configure the CI so the build will also run when tags are created:
  1. on:
  2.   push:
  3.     branches:
  4.     - master
  5.   release:
  6.     types:
  7.     - created
  1. Add a publish job to the pipeline that calls npm run deploy with the secret variable.
  1. - name: Publish
  2.   if: success() && startsWith( github.ref, 'refs/tags/releases/') && matrix.os == 'ubuntu-latest'
  3.   run: npm run deploy
  4.   env:
  5.     VSCE_PAT: ${{ secrets.VSCE_PAT }}

The if property tells the CI to run the publish step only in certain cases.

In our example, the condition has three checks:

  • success() - Publish only if the tests pass.
  • startsWith( github.ref, 'refs/tags/releases/') - Publish only if a tagged (release) build.
  • matrix.os == 'ubuntu-latest' - Include if your build runs on multiple agents (Windows, Linux, etc.). If not, remove that part of the condition.

Travis CI

vscode-test also includes a Travis CI build definition. The way to define environment variables in Travis CI is different from other CI frameworks, so the xvfb script is also different:

  1. language: node_js
  2. os:
  3. - osx
  4. - linux
  5. node_js: 10
  7. install:
  8. - |
  9.   if [ $TRAVIS_OS_NAME == "linux" ]; then
  10.     export DISPLAY=':99.0'
  11.     /usr/bin/Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
  12.   fi
  13. script:
  14. - |
  15.   echo ">>> Compile vscode-test"
  16.   yarn && yarn compile
  17.   echo ">>> Compiled vscode-test"
  18.   cd sample
  19.   echo ">>> Run sample integration test"
  20.   yarn && yarn compile && yarn test
  21. cache: yarn

Travis CI automated publishing

  1. Set up VSCE_PAT as an encrypted secret using the Travis CI encryption key usage instructions.
  2. Install vsce as a devDependencies (npm install vsce --save-dev or yarn add vsce --dev).
  3. Declare a deploy script in package.json without the PAT.
  1. "scripts": {
  2.   "deploy": "vsce publish --yarn"
  3. }
  1. Add an after_script stage to the job that calls npm run deploy with the secret variable.
  1. after_script:
  2. - |
  3.   echo ">>> Publish"
  4.   yarn deploy
  6. stages:
  7. - name: after_script
  8.   if: env(TRAVIS_TAG) =~ ^v

The stages property tells the CI to include stages when certain conditions are met.

In our example, the condition has one check:

  • env(TRAVIS_TAG) =~ ^v - Publish only if a tagged (release) build that starts with the letter v.

Common questions

Do I need to use Yarn for continuous integration?

All of the above examples refer to a hypothetical project built with Yarn, but can be adapted to use npm, Grunt, Gulp, or any other JavaScript build tool.