Deployment structure
We currently have different types of servers.
- Production. Can contain GDPR data and must be as online as possible. Examples are: DFG, PKD, mainnet, testnet, Browser.
- Demo. Can contain GDPR data (however only if there is a DPA) and must be as online as possible, will be locked to concrete versions. Examples are: ZilNet, whitelabel VPR, HTS.
- Backup. Contains copies of databases and files from production systems.
- Test. Used for temporary tests and staging, this environment can be expected to be promoted to production. This environment cannot be expected to be always online nor can it contain GDPR data. Examples are: VPR, DFG, ICRC.
- Runners for CI. Used in executing pipelines.
Deployment
First time setup
Read https://gitlab.com/secata/server-config/-/blob/main/readme.md
- Log in to the deployment server, naja:
ssh -A naja - From there SSH to each server to add those to known hosts. For each server:
ssh *ip*, press yes, logout. - Log out from naja.
Deploying
We deploy using Ansible through naja.
From your prompt run:
ssh -A naja 'cd server-config && git pull && ./deploy-services.sh'
When this is executed on naja, it gives the possibility to log who has deployed and when.
We deploy from main, this means that any MR merged to main immediately allows deployment. See the developer guide for further details.
Changes in the production environment cannot be done outside ansible.
Force majeure
Force majeure dictates that two people are watching the terminal as changes are made. Force majeure should always be the exception.
Monitoring
We monitor our production sites using uptime robot:
If a service is down, it alerts every developer. The person responsible is required to let everyone else know that the issue is being handled. This is done either in a mail to all developers or in the operations chat. Email is preferred outside of normal work hours. In case of planned downtime, the responsible should pause the monitor to avoid unnecessary alerts.