Skip to main content

2 posts tagged with "devops"

View All Tags

· 5 min read
TheBidouilleur

Introduction à Nix et NixOS

NixOS est une distribution Linux sortie initialement en 2003 et créée par (Eelco Dolstra, un chercheur travaillant sur la création d'un système immuable. Celle-ci se base sur le gestionnaire de paquet Nix qui permet de gérer la configuration du système à l'aide de fichiers Nix (un langage de programmation similaire au Haskell).

Ainsi si je souhaite créer un utilisateur kiko sur mon système, je peux écrire ceci dans mon fichier configuration.nix (qui est le fichier de configuration de l'OS initial)

  users.users.kiko = {
isNormalUser = true;
description = "kiko";
extraGroups = [ "networkmanager" "wheel" "sudo" ];
packages = with pkgs; [
firefox
vim
neovim
kubectl
terraform
];
};

Ou si je veux installer des programmes dans le système, je peux écrire ça :

  environment.systemPackages = with pkgs; [
vim
wget
htop
];

Et là, si vous êtes habitués aux gestionnaires de paquets normaux : vous avez remarqué que mon utilisateur peut installer des librairies de manière autonome.

C'est l'un des points forts de Nix ! Des environnements éphémères, des librairies contradictoires qui peuvent cohabiter, et des utilisateurs entièrements indépendants.

Nix (nous parlons du gestionnaire de paquets) autorise chaque utilisateur à avoir son propre PATH (ex: /run/wrappers/bin:/home/kiko/.nix-profile/bin:/etc/profiles/per-user/kiko/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin ). Mais il ne se limite pas qu'à ça : on peut créer des environnements temporaires assez rapidement pour ne pas avoir à installer un programme et pouvoir s'en servir ponctuellement.

❯ cowsay
The program 'cowsay' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
nix-shell -p cowsay
nix-shell -p neo-cowsay

~
❯ nix-shell -p cowsay
this path will be fetched (0.01 MiB download, 0.05 MiB unpacked):
/nix/store/9647mfqndy0aa8qkniqa05qc9yi575ny-cowsay-3.04
copying path '/nix/store/9647mfqndy0aa8qkniqa05qc9yi575ny-cowsay-3.04' from 'https://cache.nixos.org'...

~ via ❄️ impure (shell)
❯ cowsay "J aime la bidouille"
_____________________
< J aime la bidouille >
---------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

~ via ❄️ impure (shell)
exit
exit

~ took 44s
❯ cowsay
The program 'cowsay' is not in your PATH. It is provided by several packages.
You can make it available in an ephemeral shell by typing one of the following:
nix-shell -p cowsay
nix-shell -p neo-cowsay

Nous avons créé un environnement similaire au notre.. mais avec le binaire cowsay.

Mais nous avons parlé d'environnement, alors créons un réel nix-shell plus complet...

Nix-Shell

Créons 2 fichiers.

# default.nix
with (import <nixpkgs> {});
let
my-python-packages = python-packages: with python-packages; [
requests
];
python-with-my-packages = python3.withPackages my-python-packages;
in
mkShell {
buildInputs = [
python-with-my-packages
];
}
# app.py
import requests
response = requests.get('http://perdu.com')
print(response.content)

Le fichier app.py est notre très complèxe application tandis que le fichier default.nix décrit l'environnement requis. Si je lance directement l'application python3 app.py je me retrouve avec une erreur car je n'ai pas installé python3 dans mon environnement.. Je peux créer un nix-shell avec python et lancer mon app.py :

❯ nix-shell -p python38 # création d'un env avec python3.8

/tmp/python via 🐍 v3.8.13 via ❄️ impure (shell)
❯ python3 app.py
Traceback (most recent call last):
File "app.py", line 1, in <module>
import requests
ModuleNotFoundError: No module named 'requests'

Mais nous nous retrouvons avec un autre problème de dépendance.. Il est donc possible de créer notre environnement à l'aide du fichier default.nix qui contiendra Python et la librairie requests (indispensable pour notre application.

Par défaut, nix-shell va chercher les fichiers nommés default.nix dans notre répertoire courant.

/tmp/python via 🐍 took 5m28s 
❯ nix-shell

/tmp/python via 🐍 v3.9.13 via ❄️ impure (nix-shell)
❯ python3 app.py
b"<html><head><title>Vous Etes Perdu ?</title></head><body><h1>Perdu sur l'Internet ?</h1><h2>Pas de panique, on va vous aider</h2><strong><pre> * <----- vous &ecirc;tes ici</pre></strong></body></html>\n"

Avec cette méthode, il est possible d'avoir plusieurs environnements pour lancer des applications différentes sans se soucier des effets de bords sur nos autres programmes.

Home-Manager

Si jamais je résume ce que nous avons vu :

  • Comment NixOS peut automatiser une configuration d'OS (On pourrait voir comment l'installer de la même manière)
  • Comment créer des environnements indépendants Il reste un aspect essentiel au passage sur Nix: le déploiement d'une configuration utilisateur !

Nix permet de déployer bien plus que quelques programmes, il existe une librairie d'instruction pour faciliter la configuration/déploiement d'un logiciel. Par exemple, à la création de mon utilisateur quotidien, je dois parametrer Git avec mon nom, et mon mail avant chaque commit..

git config --global user.name "Toto"
git config --global user.email "toto@toto.com"

Avec Nix, je peux créer un fichier dans mon home et garder cette configuration en dur :

  programs = {
git = {
enable = true;
userName = "Toto";
userEmail = "toto@toto.com";
ignores = [
"*~"
"*.swp"
];
};
};

J'ai donc plusieurs fichiers Nix me permettant d'installer mes programmes, de configurer Git, d'installer mon EMacs-Doom avec mes paramètres, de déposer mes dotfiles aux bons endroits. Pour l'instant.. ma configuration est publique et disponible ici, à voir ce que j'en ferai à l'avenir.. :)

Conclusion

Nix est vaste, très vaste, et il peut être compliqué d'en apprendre les bases. La communauté est au courant du manque de documentation et fait beaucoup d'effort pour donner une image agréable à Nix pour les débutants. Je pense que Nix a un potentiel non-négligeable pour les workstations/serveurs et pourrait même remplacer des outils de déploiement d'OS comme Packer.

On peut trouver les instructions / packages sur l'incroyable site search.nix.org. Je pense continuer à apprendre Nix jusqu'à pouvoir moi-même contribuer à la communauté et maintenir mes propres packages.

Seule complexité reste d'apprendre le langage Nix ! Mais avez-vous entendu parler de Guix.. ?

· 5 min read
TheBidouilleur

[ This article is from my old-blog, it will also be available in the "Documentation" section of the site ]

Introduction

Soon 7 years since my main infrastructure is on Proxmox. It's the hypervisor I trust most, which is also free and open-source. As soon as I have to deploy more than 2 virtual machines and can choose the environment: Proxmox will be my first choice. It offers a complete and efficient webui, without forgetting the advantage of command line tools. I don't rule out that someday, I may change my environment. And today, I have new needs in my hypervisor: Automate a complete deployment of my infrastructure, and since I will not reinstall each machine individually, I must start from a "base" that will serve as a template for the machine system to be pre-configured as I wish. And this famous template, I can make by hand.... or I can deploy it automatically with Packer!

What is Packer?

Packer is a tool developed by hashicorp (a company that provides open-source programs in the world of devops) allowing to deploy a template virtual machine automatically. In a practical case, Packer will connect to your public-cloud(aws, oracle, scaleway) / hypervisor(proxmox, qemu, esxi) to send instructions to install the virtual machine. (Like resources needed. RAM, CPU cores, type of bios)

How does Packer work?

Packer has few dependencies, it needs a public hypervisor/cloud, access to the "screen"* of the virtual machine, and ssh access for Packer to verify that the installation has completed (and also to launch a config management tool such as ansible).

*virtual machine screen will be used to send keystrokes.

A little vocabulary

The place where Packer deploys the VM is called Builder, in my case: It's Proxmox! And the term "provisioning" refers to the tool that will finish configuring the VM (Ex: Ansible).

Create our first template

Before we tackle a big fish like debian, we'll start with a simpler system to install: Alpine. The alpine installer will ask about ten questions, one by one. There is an answer file system that will automatically answer questions, but I could not run this file under alpine. (Only on Alpine, the answer file works on debian).

As I cannot use an answer file: we will answer questions manually (by sending keystrokes).

/!\ In the rest of this article, I will base myself on this deposit that is hosted on my gitea: packer-alpine-proxmox.

{
"description": "Build Alpine Linux 3 x86_64 Proxmox template",
"variables": {

"proxmox_url": "{{env `proxmox_url`}}",
"proxmox_username":"{{env `proxmox_user`}}",
"proxmox_password": "{{env `proxmox_password`}}",
"proxmox_host": "{{env `proxmox_node`}}",

"storage_name": "{{env `storage_name`}}",
"bridge": "{{env `bridge`}}",
"vm_id": "9001",
"vm_name": "alpine3-tf",
"template_description": "Alpine Linux 3.11 x86_64 template built with packer",
"vm_memory": "1024",

"ssh_username": "root",
"ssh_password": "{{env `ssh_password`}}"
},
"sensitive-variables": ["proxmox_password", "ssh_password" ],
"provisioners": [
{
"type": "ansible",
"playbook_file": "./playbook/provisioning.yml",
"ansible_env_vars": ["ANSIBLE_FORCE_COLOR=True" ]
}
],
"builders": [
{
"type": "proxmox",
"proxmox_url": "{{user `proxmox_url`}}",
"insecure_skip_tls_verify": true,
"username": "{{user `proxmox_username`}}",
"password": "{{user `proxmox_password`}}",
"vm_id": "{{user `vm_id`}}",
"vm_name": "{{user `vm_name`}}",
"template_description":"{{user `template_description`}}",
"memory": "{{user `vm_memory`}}",
"cores": "2",
"os": "l26",
"http_directory": "http",

"node": "{{user `proxmox_host`}}",
"network_adapters": [
{
"model": "virtio",
"bridge": "{{user `bridge`}}"
}
],
"disks": [
{
"type": "virtio",
"disk_size": "16G",
"storage_pool": "{{user `storage_name`}}",
"storage_pool_type": "directory",
"format": "qcow2"
}
],
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_timeout": "15m",
"ssh_certificate_file": "/root/id_rsa",
"iso_file": "{{user `storage_name`}}:iso/alpine-virt-3.15.0-x86_64.iso",
"unmount_iso": true,
"boot_wait": "15s",
"boot_command": [
"<wait25>root<enter><wait4>",
"setup-alpine<enter><wait8>",
"<enter><wait4>",
"alpine-tf<enter><wait4><enter><wait4>",
"dhcp<enter>",
"<wait5>n<enter><wait5>",
"{{user `ssh_password`}}<enter><wait5>",
"{{user `ssh_password`}}<enter><wait>",
"<wait5>",
"Europe/Paris <enter><wait2><enter><wait5>",
"n<enter>",
"<wait1>1<enter><wait3>",
"<enter><wait2>",
"vda<enter>",
"lvm<enter>",
"sys<enter>",
"<wait2>",
"y<enter><wait35>",


"reboot <enter>",
"<wait65>",

"root<enter><wait8>",
"{{user `ssh_password`}}<enter><wait5> ",
"<wait10>",

"apk update && apk add curl<enter>",
"mkdir -p ~/.ssh<enter>",
"touch ~/.ssh/authorized_keys<enter><wait5>chmod 600 ~/.ssh/authorized_keys<enter><wait5>",
"curl http://{{ .HTTPIP }}:{{ .HTTPPort }}/authorized_keys >> ~/.ssh/authorized_keys<enter>",
"echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config <enter>",
"<wait2>service sshd restart <enter> <wait2>",
"curl http://{{ .HTTPIP }}:{{ .HTTPPort }}/repositories > /etc/apk/repositories<enter>",
"<wait>apk update <enter>",

"apk add python3<enter><wait1>",
"curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py<enter> <wait2>",
"python3 /tmp/get-pip.py <enter> <wait2>",

"apk add qemu-guest-agent<enter><wait3>",
"rc-update add qemu-guest-agent<enter>",
"service qemu-guest-agent start<enter>"

]
}
]
}

We'll quickly go through the structure of this Packer file:

  • the part "Variable" relates to static variables and/or environment variables (We'll see later)
  • Provision designates the command to be launched after the creation of the template
  • and what concerns the template itself (parameters, hypervisors...) is in the builder part

and the boot_command part in Builder is the list of all keyboard entries that Packer will type, It often places the download of Packer's Preseed to the VM.

small explanation about file transfer from packer to template: Packer, at start, will create a web server with the contents of the http/ folder, if you place files inside it, you can tell packer to type the following command to recover files. (Ex: Preseed, ssh keys etc.)

curl  http://{{ .HTTPIP }}:{{ .HTTPPort }}/fichier

So, in this Alpine installation, I will answer questions one by one, with pre-configured timers (which count in seconds). And at the end of the installation, we launch the playbook provisioning.yml which allows me to install the dependencies useful to my VMs. There is no need to go further in the playbook: it's still a template.

Why running packer through shell script?

If you went to my repository (linked above), you probably saw the file buid.sh.

#!/bin/bash
#export ssh_password=$(vault kv get -field root_password secrets/password)
export proxmox_password=$(vault kv get -field proxmox_password kv/wysux)
export proxmox_user=$(vault kv get -field proxmox_user kv/wysux)
export proxmox_node=$(vault kv get -field proxmox_node kv/wysux)
export proxmox_url=$(vault kv get -field proxmox_url kv/wysux)

export ssh_password="toto13"

export bridge="vmbr0"
export storage_name="local"

rm http/authorized_keys || true
for f in ssh/*.pub; do
name_of_key=$(echo $f | cut -d "/" -f2 )
echo -e "#$name_of_key" >> http/authorized_keys
key=$(cat $f)
echo -e "$key" >> http/authorized_keys
done
packer build alpine-3-amd64-proxmox.json

This file will provide some parameters to Packer such as variables containing passwords. I use Vault to retrieve sensitive information from a remote server. You can choose to not use Vault by placing passwords directly in clear text.