Developing the Data Platform
There is a lot of technical tooling we use when developing the Analytical and Data Platforms.
To ease the onboarding process, we have created a dev container that contains all the tools you need to get up and running.
This implementation of the dev container is designed to have repositories cloned into it under the persistent workspace /home/vscode/workspace.
Once we begin our journey to the monorepo, this should become redundant, but that means that right now, it is not intended to use this with GitHub Codespaces.
Getting Started
Prerequisites
macOS
Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"1Password
You will need to request access to 1Password via Operations Engineering.
brew install 1passwordDocker
If you are have an Apple Silicon Mac, you will need to enable Rosetta emulation, see Docker’s documentation for more information.
brew install --cask dockerNode
brew install nodeDev Container CLI
sudo npm install --location=global @devcontainers/cli@latestVisual Studio Code
brew install --cask visual-studio-codeDev Containers Visual Studio Code Extension
/Applications/Visual\ Studio\ Code.app/Contents/Resources/app/bin/code --install-extension ms-vscode-remote.remote-containers
Setup
Your GPG and SSH agents are mounted from macOS, so GPG and SSH commands inside the container should work without any configuration updates
We recommend using 1Password to manage your SSH keys, and this document will reflect that
- Follow 1Password’s instructions on how to setup SSH keys and their SSH agent (link)
If you already have an existing key, you can add it to 1Password (link) and skip to step 3 of 1Password’s instructions.
Permanently set
SSH_AUTH_SOCKby adding the following to your~/.zshrcor~/.bashrcfileexport SSH_AUTH_SOCK=~/Library/Group\ Containers/2BUA8C4S2C.com.1password/t/agent.sockClone the Data Platform repository
Example mkdir -p ~/Developer/github.com/ministryofjustice git clone git@github.com:ministryofjustice/data-platform.git ~/Developer/github.com/ministryofjustice/data-platformOpen the Data Platform repository in Visual Studio Code
Example cd ~/Developer/github.com/ministryofjustice/data-platform /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/bin/code .Open the Visual Studio Code Command Palette using
Shift-Command-PRun
Dev Container: Reopen in ContainerWait for the container to build, it may take 5 to 10 minutes
Open a new terminal window in Visual Studio Code using
Control-Shift-BacktickClone the Data Platform repository (this time its inside the container)
Example mkdir --parents ${HOME}/workspace/github.com/ministryofjustice git clone git@github.com:ministryofjustice/data-platform.git ${HOME}/workspace/github.com/ministryofjustice/data-platform
Updating
When new functionality is added you will need to rebuild the container, to do this:
Pull the latest changes from the Data Platform repository on the host machine
Example cd ~/Developer/github.com/ministryofjustice/data-platform git fetch git pullOpen the Data Platform repository in Visual Studio Code
Example cd ~/Developer/github.com/ministryofjustice/data-platform /Applications/Visual\ Studio\ Code.app/Contents/Resources/app/bin/code .Open the Visual Studio Code Command Palette using
Shift-Command-PRun
Dev Container: Rebuild and Reopen in Container
Tools
Tools are bundled into features that focus on one area of functionality, e.g. AWS, Kubernetes, and Terraform
You can find the full feature list here, but documentation on a select few can be found below
Pre-Commit
We use pre-commit to run checks on our code before it is committed
Once you’ve cloned the Data Platform repository, you can install the pre-commit hooks using the following command:
pre-commit install
EditorConfig
We use EditorConfig to maintain consistent coding styles between different editors and IDEs
AWS Vault
Our AWS accounts are accessed using AWS IAM Identity Center. A curated AWS configuration file is provided as part of the AWS feature
You can see the configuration file here
In its current form, it only offers AdministratorAccess level access (if you have access to it), but we hope to add more granular access in the future
Kubernetes
We host our own EKS clusters in the Analytical and Data Platforms. A curated Kubernetes configuration file is provided as part of the Kubernetes feature
You can see the configuration file here
The provided configuration file handles authentication to EKS clusters using AWS Vault.
Cloud Platform Authentication
Rebuilding the dev container will reset the Kubernetes configuration file, so you’ll need to authenticate again.
Login to Cloud Platform Kuberos (https://login.cloud-platform.service.justice.gov.uk)
Obtain the command from “Authenticate Manually”
Replace your Auth0 email with the generic
auth0userkubectl config set-credentials "auth0" \ --auth-provider=oidc \ --auth-provider-arg=client-id="..." \ --auth-provider-arg=client-secret="..." \ --auth-provider-arg=id-token="..." \ --auth-provider-arg=refresh-token="..." \ --auth-provider-arg=idp-issuer-url="https://justice-cloud-platform.eu.auth0.com/"Set Cloud Platform to the current context
kubectl config use-context live.cloud-platform.service.justice.gov.uk
Developing a Feature
Testing
Run the script
bash scripts/devcontainer/feature-test.sh ${FEATURE_NAME}
Debugging
Build the base image
docker build --file .devcontainer/features/test/Dockerfile --tag devcontainer .Run the container
docker run -it --rm \ --volume $( pwd ):/workspace \ --volume $( pwd )/.devcontainer/features/src/base/devcontainer-utils:/usr/local/bin/devcontainer-utils \ devcontainerRun the script
bash -x /workspace/.devcontainer/features/src/<feature>/install-<tool>.sh
devcontainer Settings
dotfiles
You can populate your devcontainer with personal dotfiles by adding them to a personal public repository called dotfiles.
An example of this can be seen here. This configuration is defined here.