Automatically building, previewing, and pushing your book with CircleCI

CircleCI is a continuous integration service that lets you run various commandsevery time a new change is made to a repository. This can be used to build your book,preview changes, and even push live HTML as you update your book content.

In order to accomplish each of these, we'll use a CircleCI configuration file. Thisis a YAML file that is used to tell Circle what to do with your repository.

In each case, the expectation is that your master branch holds your book content.

We'll step through each piece of a sample CircleCI configuration to show you howto accomplish this.

Preparing CircleCI

First of all, you should set up your CircleCI account to start running CI jobs foryour book repository. Follow these steps:

  1. https://circleci.com/gh/{{YOUR-GITHUB-ORG}}/{{YOUR-GITHUB-REPO}}/edit#advanced-settings

Find the "build forked pull requests" section, and switch it to ON.

Now, CircleCI will start watching your repository. If it finds a Circle configurationfile (more information on this below), it'll run a CI job according to the configurationit finds.

You can copy/paste the empty CircleCI configuration here. This won't actually do anythingbut we'll add to it later:

  1. # Tell CircleCI which version of its API you wish to use
  2. version: 2.1
  3. jobs:
  4.     # Jobs define the different parts of your CircleCI workflow
  5.     # They can depend on one another, use pieces from one another, etc.
  6.     # We'll fill them in later
  7. workflows:
  8.     # Workflows tell CircleCI the order in which to run your jobs
  9. commands:
  10.     # Commands are re-usable chunks of steps that can be shared across jobs

Let's start filling out this template with a few jobs. In each case, we'lluse a Python Docker image to both build each page's HTML, and build the book'sHTML.

Step 1: Build each page's HTML

First you'll build each page's HTML. This is the initial conversion fromipynb, md, etc files. We'll use a Python container for this in order touse the Jupyter Book command-line interface with jupyter-book build.

You can build your book's HTML files and preview them using CircleCI artifacts.To do this, you'll need to use two CircleCI jobs:

We'll need to persist the results of this step so that they are available insubsequent steps. Here's the CircleCI configuration that will accomplish this, which you canadd to the skeleton configuration you've created above:

  1. jobs:
  2.   build_page_files:
  3.     docker:
  4.       - image: circleci/python:3.6-stretch
  5.     steps:
  6.       # Get our data and merge with upstream
  7.       - checkout
  8.  
  9.       # Install the packages needed to build our documentation
  10.       # This will depend on your particular package!
  11.       - run: pip install --user -r requirements.txt
  12.  
  13.       # Build the page intermediate HTML files
  14.       - run:
  15.           name: Build page intermediate HTML files
  16.           command: jupyter-book build .
  17.  
  18.       # Persist the specified paths to be used in subsequent jobs
  19.       # (see https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs)
  20.       - persist_to_workspace:
  21.           root: .
  22.           paths:
  23.             - ./_build/

Note that, at the end of this job, we've persisted the contents of the _build/ folder.This allows us to re-use these contents in subsequent jobs.

Step 2: Install Jekyll with Anaconda

Now that our page HTML files have been built, we can use Jekyll to buildthe HTML for our entire book. This is useful for two purposes:

  • previewing your Jupyter Book using CircleCI artifacts.
  • publishing the HTML of your book to someplace online In both cases we need to install Jekyll and build the HTML for the book,so let's first define a CircleCI command to do this.

The following command will copy over the built page HTML from the previousjob, install Miniconda which we'll use for environment management, theninstall Jekyll and the dependencies needed to build your book's HTML usingconda-forge.

  1. commands:
  2.   prepare_jekyll_installation:
  3.     steps:
  4.       - checkout
  5.       - attach_workspace:
  6.           # Must be absolute path or relative path from working_directory
  7.           at: /tmp/workspace
  8.  
  9.       # Grab the the built intermediate files from the last step
  10.       - run:
  11.           name: Copy over built site files
  12.           command: |
  13.             rm -rf _build
  14.             cp -r /tmp/workspace/_build .
  15.  
  16.       # Install miniconda to test install
  17.       - run:
  18.           name: install miniconda
  19.           command: |
  20.             export MINICONDA=$HOME/miniconda
  21.             echo "export PATH=$MINICONDA/bin:$PATH" >> $BASH_ENV
  22.             source $BASH_ENV
  23.             hash -r
  24.             wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
  25.             bash miniconda.sh -b -f -p $MINICONDA
  26.             conda config --set always_yes yes
  27.             conda update conda
  28.             conda info -a
  29.             conda create -n testenv python=3.7.0
  30.             source activate testenv
  31.             rm miniconda.sh
  32.  
  33.       # Install Ruby/Jekyll dependencies
  34.       - run:
  35.           name: Installing Ruby/Jekyll from conda-forge
  36.           command: conda install -c conda-forge rb-github-pages
  37.  
  38.       # Build the book's HTML w/ the base_url for CircleCI artifacts
  39.       - run:
  40.           name: Install book Ruby dependencies
  41.           command: bundle install

Step 3: Build and preview your book's HTML with Circle artifacts

We'll re-use the command from above in order to preview our built siteusing Circle artifacts. Add the job below to your CircleCI configuration file.

  1. jobs:
  2.   doc:
  3.     docker:
  4.       - image: circleci/python:3.7-stretch
  5.     steps:
  6.       - prepare_jekyll_installation
  7.       - run:
  8.           name: Build the website
  9.           command: bundle exec jekyll build --baseurl /0/html/
  10.  
  11.       # Tell Circle to store the documentation output in a folder that we can access later
  12.       - store_artifacts:
  13.           path: _site/
  14.           destination: html

After the Jekyll installation command, it then runs the build command from Jekyll,which outputs all the HTML for your site. We add the —baseurl /0/html because thisis the prefix for the Jekyll artifact URL. Finally, we use the store_artifactscommand to tell Jekyll to keep these artifacts for later. Once this job completes,you'll be able to click the "Artifacts" tab to preview your book HTML.

Step 4: Automatically publish live HTML from your master branch

You can also choose to automatically push built HTML from your master branchto a live textbook. This lets you automatically deploy changes to your bookso that they go live online. Again, we'll use the page HTML command defined above.The job will be very similar to the HTML artifacts preview job, with an extrastep to actually push the book's HTML online.

This step assumes that you are hosting your live book on a Git repositoryusing GitHub Pages. We'll need the include a security key that allows CircleCI push accessto your GitHub repository. You should first create an SSH key with write access to therepository that will be hosting your live site. Then, use the below configurationto tell Circle to automatically push to this repository.

  1. jobs:
  2.   deploy:
  3.     docker:
  4.       - image: circleci/python:3.7-stretch
  5.     steps:
  6.       # Add deployment key fingerprint for CircleCI to use for a push
  7.       - add_ssh_keys:
  8.           fingerprints:
  9.             - "{{ YOUR SSH KEY FINGERPRINT }}"
  10.  
  11.       - prepare_jekyll_installation
  12.       - run:
  13.           name: Build the website for deploy
  14.           command: bundle exec jekyll build
  15.  
  16.       # Deploy the built site with ghp-import
  17.       - run:
  18.           name: Deploying site using ghp-import
  19.           command: |
  20.             pip install ghp-import
  21.             ghp-import -p -f -n ./_site/

In this case we've used the excellent ghp-import tool for pushing to github pages.The command pushes the contents of ./_site (your book's HTML) to the gh-pagesbranch of the repository. The -n flag adds a .nojekyll file to the built HTML,which ensures that Jekyll will treat it as raw HTML.

Step 4: Tying these workflows together

Now that we've defined several jobs above, we need to tell CircleCI how touse them sequentially (or in parallel). In particular, we want the jobthat builds each page's HTML to run first sothat the each page's HTML can be stitched together into a book.Here's the configuration for this:

  1. workflows:
  2.   version: 2
  3.   default:
  4.     jobs:
  5.       - build_page_html:
  6.           filters:
  7.             branches:
  8.               ignore:
  9.                 - gh-pages
  10.       - doc:
  11.           requires:
  12.             - build_page_html
  13.           filters:
  14.             branches:
  15.               ignore:
  16.                 - gh-pages
  17.       - deploy:
  18.           requires:
  19.             - build_page_html
  20.           filters:
  21.               branches:
  22.                 only:
  23.                   - master
  24.                 ignore:
  25.                   - gh-pages

Appendix: The full configuration file

Below is a full CircleCI configuration file that covers each of the stepsabove. Note that the syntax may be slightly different because we're putting eachstep above in a single file.

  1. # NOTE: This is an example CircleCI configuration that
  2. # will build your book and preview its HTML content.
  3. # You will probably have to modify it in order to get it working
  4. # just the way you want. See https://jupyterbook.org/advanced/circleci.html
  5. # for more information
  6. version: 2.1
  7. jobs:
  8.   build_page_html:
  9.     docker:
  10.       - image: circleci/python:3.7-stretch
  11.     steps:
  12.       - checkout
  13.       - run: pip install --user -r requirements.txt
  14.       - run:
  15.           name: Build site intermediate files
  16.           command: jupyter-book build .
  17.  
  18.       # Persist the built files for the deploy step
  19.       - persist_to_workspace:
  20.           root: .
  21.           paths:
  22.             - _build/
  23.  
  24.   doc:
  25.     docker:
  26.       - image: circleci/python:3.7-stretch
  27.     steps:
  28.       - prepare_jekyll_installation
  29.       - run:
  30.           name: Build the website
  31.           command: bundle exec jekyll build --baseurl /0/html/
  32.  
  33.       # Tell Circle to store the documentation output in a folder that we can access later
  34.       - store_artifacts:
  35.           path: _site/
  36.           destination: html
  37.  
  38.   # Deploy the built site to jupyter-book.github.io
  39.   deploy:
  40.     docker:
  41.       - image: circleci/python:3.7-stretch
  42.     steps:
  43.       # Add deployment key fingerprint for CircleCI to use for a push
  44.       - add_ssh_keys:
  45.           fingerprints:
  46.             - "{{ YOUR SSH FINGERPRINT }}"
  47.  
  48.       - prepare_jekyll_installation
  49.       - run:
  50.           name: Build the website for deploy
  51.           command: bundle exec jekyll build
  52.  
  53.       # Deploy the built site with ghp-import
  54.       - run:
  55.           name: Deploying site using ghp-import
  56.           command: |
  57.             pip install ghp-import
  58.             ghp-import -p -f -n ./_site/
  59.  
  60.  
  61. # Tell CircleCI to use this workflow when it builds the site
  62. workflows:
  63.   version: 2
  64.   default:
  65.     jobs:
  66.       - build_page_html:
  67.           filters:
  68.             branches:
  69.               ignore:
  70.                 - gh-pages
  71.       - doc:
  72.           requires:
  73.             - build_page_html
  74.           filters:
  75.             branches:
  76.               ignore:
  77.                 - gh-pages
  78.       - deploy:
  79.           requires:
  80.             - build_page_html
  81.           filters:
  82.               branches:
  83.                 only:
  84.                   - master
  85.                 ignore:
  86.                   - gh-pages
  87.  
  88. commands:
  89.   prepare_jekyll_installation:
  90.     steps:
  91.       - checkout
  92.       - attach_workspace:
  93.           # Must be absolute path or relative path from working_directory
  94.           at: /tmp/workspace
  95.  
  96.       # Grab the the built intermediate files from the last step
  97.       - run:
  98.           name: Copy over built site files
  99.           command: |
  100.             rm -rf ./_build
  101.             cp -r /tmp/workspace/_build .
  102.  
  103.       # Install miniconda to test install
  104.       - run:
  105.           name: install miniconda
  106.           command: |
  107.             export MINICONDA=$HOME/miniconda
  108.             echo "export PATH=$MINICONDA/bin:$PATH" >> $BASH_ENV
  109.             source $BASH_ENV
  110.             hash -r
  111.             wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
  112.             bash miniconda.sh -b -f -p $MINICONDA
  113.             conda config --set always_yes yes
  114.             conda update conda
  115.             conda info -a
  116.             conda create -n testenv python=3.7.0
  117.             source activate testenv
  118.             rm miniconda.sh
  119.  
  120.       # Install Ruby/Jekyll dependencies
  121.       - run:
  122.           name: Installing Ruby/Jekyll from conda-forge
  123.           command: conda install -c conda-forge rb-github-pages
  124.  
  125.       # Build the book's HTML w/ the base_url for CircleCI artifacts
  126.       - run:
  127.           name: Install book Ruby dependencies
  128.           command: bundle install

This page was created by The Jupyter Book Community