• Using the forked version of Docker Machine
  • Using GPUs on Google Compute Engine

The auto scale feature was introduced in GitLab Runner 1.1.0.

Docker uses containers to create virtual environments that isolate a TensorFlow installation from the rest of the system. TensorFlow programs are run within this virtual environment that can share resources with its host machine (access directories, use the GPU, connect to the Internet, etc.). If you need to use any virtualization/cloud providers that aren’t handled by Docker Machine’s internal drivers, the appropriate driver plugin must be installed. The Docker Machine driver plugin installation and configuration is out of the scope of this documentation. For more details please read the Docker Machine documentation. Depending on the instance type, you can either download a public NVIDIA driver, download a driver from Amazon S3 that is available only to AWS customers, or use an AMI with the driver pre-installed. To install AMD drivers on an instance with an attached AMD GPU, such as a G4ad instance, see Install AMD drivers on Linux instances instead. Step 2 – Install Nvidia-Docker. NVIDIA designed NVIDIA-Docker in 2016 to enable portability in Docker images that leverage NVIDIA GPUs. It wrapped CUDA drivers for ease of use for Docker with a GPU. Its main function is to mount the user mode components of the driver, and the GPU device files into the container at launch. Mac os 10.7 dmg.

Make sure you have installed the NVIDIA driver and Docker engine for your Linux distribution Note that you do not need to install the CUDA Toolkit on the host system, but the NVIDIA driver needs to be installed. For instructions on getting started with the NVIDIA Container Toolkit, refer to the installation guide.

For an overview of the autoscale architecture, take a look at thecomprehensive documentation on autoscaling.

Forked version of Docker machine

Because docker-machine is in maintenancemode, GitLab isproviding it’s own fork ofdocker-machine,which is based on the latest master branch of docker-machine withsome additional patches for the following bugs:

The intent of this fork is to fix critical and bugs affecting runningcosts only. No new features will be added.

Preparing the environment

To use the autoscale feature, Docker and GitLab Runner must beinstalled in the same machine: Mac os 32 bit.

1. Log in to a new Linux-based machine that will serve as a bastion serverwhere Docker will spawn new machines from
2. Optionally but recommended, prepare aproxy container registry and a cache serverto be used with the autoscaled runners

If you need to use any virtualization/cloud providers that aren’t handled byDocker Machine’s internal drivers, the appropriate driver plugin must beinstalled. The Docker Machine driver plugin installation and configuration isout of the scope of this documentation. For more details please read theDocker Machine documentation

Configuring GitLab Runner

1. Register a runner and select thedocker+machine executor when asked.
2. Edit config.toml and configurethe runner to use Docker machine. Visit the dedicated page covering detailedinformation about GitLab Runner Autoscaling.

3. The first time you’re using Docker Machine, it’s best to execute manuallydocker-machine create .. with your chosen driver and all options from theMachineOptions section. This will set up the Docker Machine environmentproperly and will also be a good validation of the specified options.After this, you can destroy the machine with docker-machine rm [machine_name]and start the runner.

   Multiple concurrent requests to docker-machine create that are doneat first usage are not good. When the docker+machine executor is used,the runner may spin up few concurrent docker-machine create commands. IfDocker Machine was not used before in this environment, each started processtries to prepare SSH keys and SSL certificates (for Docker API authenticationbetween GitLab Runner and Docker Engine on the autoscaled spawned machine), and theseconcurrent processes are disturbing each other. This can end with a non-workingenvironment. That’s why it’s important to create a test machine manually thevery first time you set up GitLab Runner with Docker Machine.
4. Now, you can try and start a new pipeline in your project. In a few seconds,if you run docker-machine ls you should see a new machine being created.

Upgrading GitLab Runner

1. Check if your operating system is configured to automatically restart GitLabRunner (for example, by checking its service file):

   • if yes, ensure that service manager is configured to use SIGQUITand use the service’s tools to stop the process:

   • if no, you may stop the process manually:

   Sending the SIGQUIT signal will make theprocess stop gracefully. The process will stop accepting new jobs, and will exitas soon as the current jobs are finished.

2. Wait until GitLab Runner exits. You can check its status with gitlab-runner statusor await a graceful shutdown for up to 30 minutes with:

3. You can now safely install the new version of GitLab Runner without interrupting any jobs.

Using the forked version of Docker Machine

Install Nvidia Drivers Docker

Install

1. Download the appropriate docker-machine binary.Copy the binary to a location accessible to PATH and make itexecutable. For example, to download and install v0.16.2-gitlab.11:

Using GPUs on Google Compute Engine

Introduced in GitLab Docker Machine 0.16.2-gitlab.10 and GitLab Runner 13.9.

You can use the Docker Machine fork to create Google Compute Engine instanceswith graphics processing units (GPUs).GitLab Runner 13.9 is required for GPUs to work in a Docker executor.

Docker Machine GPU options

These arguments map to command-line arguments for gcloud compute.See the Google documentation on creating VMs with attached GPUsfor more details.

Verifying Docker Machine options

To prepare your system and test that GPUs can be created with Google Compute Engine:

1. Set up the Google Compute Engine driver credentialsfor Docker Machine. You may need to export environment variables to therunner if your VM does not have a default service account. Howthis is done depends on how the runner is launched. For example:

   • Via systemd or upstart: See the documentation on setting custom environment variables.
   • Via Kubernetes with the Helm Chart: Update the values.yaml entry.
   • Via Docker: Use the -e option (for example, docker run -e GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json gitlab/gitlab-runner).

2. Verify that docker-machine can create a virtual machine with yourdesired options. For example, to create an n1-standard-1 machinewith a single NVIDIA Telsa P4 accelerator, substitutetest-gpu with a name and run:

3. To verify the GPU is active, SSH into the machine and run nvidia-smi:

4. Remove this test instance to save money:

Configuring GitLab Runner

1. Once you have verified these options, configure the Docker executorto use all available GPUs in the runners.docker configuration.Then add the Docker Machine options to your MachineOptions settings in the GitLab Runner runners.machine configuration. For example:

Help & feedback

Get Help

If you didn't find what you were looking for,search the docs.

If you want help with something specific and could use community support,post on the GitLab forum.

For problems setting up or using this feature (depending on your GitLabsubscription).

Docker Install Nvidia Drivers