惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Microsoft Azure Blog
Microsoft Azure Blog
Engineering at Meta
Engineering at Meta
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
F
Fortinet All Blogs
博客园 - 叶小钗
T
Tailwind CSS Blog
Security Archives - TechRepublic
Security Archives - TechRepublic
T
The Exploit Database - CXSecurity.com
Blog — PlanetScale
Blog — PlanetScale
T
Tenable Blog
人人都是产品经理
人人都是产品经理
D
DataBreaches.Net
A
Arctic Wolf
P
Proofpoint News Feed
S
SegmentFault 最新的问题
C
CERT Recently Published Vulnerability Notes
T
Threatpost
Y
Y Combinator Blog
WordPress大学
WordPress大学
L
LINUX DO - 热门话题
酷 壳 – CoolShell
酷 壳 – CoolShell
V
Vulnerabilities – Threatpost
AWS News Blog
AWS News Blog
小众软件
小众软件
U
Unit 42
云风的 BLOG
云风的 BLOG
美团技术团队
S
Securelist
C
Cybersecurity and Infrastructure Security Agency CISA
L
LangChain Blog
G
GRAHAM CLULEY
P
Proofpoint News Feed
I
Intezer
Security Latest
Security Latest
L
Lohrmann on Cybersecurity
NISL@THU
NISL@THU
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
The Hacker News
The Hacker News
Cisco Talos Blog
Cisco Talos Blog
T
The Blog of Author Tim Ferriss
H
Heimdal Security Blog
T
Tor Project blog
MongoDB | Blog
MongoDB | Blog
The Cloudflare Blog
T
Troy Hunt's Blog
Know Your Adversary
Know Your Adversary
博客园 - 司徒正美
Google Online Security Blog
Google Online Security Blog
V
V2EX
Application and Cybersecurity Blog
Application and Cybersecurity Blog

Ivan on Containers, Kubernetes, and Server-Side

A grounded take on agentic coding for production environments Server-Side Playgrounds Reimagined: Build, Boot, and Network Your Own Virtual Labs [not a] Kubernetes 101 - Pods, Deployments, and Services As an Attempt To Automate Age-Old Infra Patterns JavaScript or TypeScript? How To Benefit From the Dichotomy On Software Design... and Good Writing Building a Firecracker-Powered Course Platform To Learn Docker and Kubernetes How To Publish a Port of a Running Container What Actually Happens When You Publish a Container Port A Visual Guide to SSH Tunnels: Local and Remote Port Forwarding Debugging Containers Like a Pro Docker: How To Debug Distroless And Slim Containers How To Extract Container Image Filesystem Using Docker | iximiuz Labs In Pursuit of Better Container Images: Alpine, Distroless, Apko, Chisel, DockerSlim, oh my! How To Start Programming In Go: Advice For Fellow DevOps Engineers Kubernetes Ephemeral Containers and kubectl debug Command How To Develop Kubernetes CLIs Like a Pro Docker Container Commands Explained: Understand, Don't Memorize | iximiuz Labs Learning Docker with Docker - Toying With DinD For Fun And Profit How To Extend Kubernetes API - Kubernetes vs. Django The Influence of Plumbing on Programming How To Call Kubernetes API from Go - Types and Common Machinery How To Call Kubernetes API using Simple HTTP Client Kubernetes API Basics - Resources, Kinds, and Objects OpenFaaS - Run Containerized Functions On Your Own Terms Learning Containers From The Bottom Up Docker Containers vs. Kubernetes Pods - Taking a Deeper Look | iximiuz Labs Learn-by-Doing Platforms for Dev, DevOps, and SRE Folks How HTTP Keep-Alive can cause TCP race condition How to Work with Container Images Using ctr | iximiuz Labs Multiple Containers, Same Port, no Reverse Proxy... Exploring Go net/http Package - On How Not To Set Socket Options DevOps, SRE, and Platform Engineering My Choice of Programming Languages Prometheus Is Not a TSDB How to learn PromQL with Prometheus Playground Prometheus Cheat Sheet - Basics (Metrics, Labels, Time Series, Scraping) Rust - Writing Parsers With nom Parser Combinator Framework pq - parse and query log files as time series Prometheus Cheat Sheet - Moving Average, Max, Min, etc (Aggregation Over Time) Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching) The Need For Slimmer Containers Understanding Rust Privacy and Visibility Model Bridge vs. Switch: Takeaways from a Real Data Center Tour | iximiuz Labs From LAN to VXLAN: Networking Basics for Non-Network Engineers | iximiuz Labs KiND - How I Wasted a Day Loading Local Docker Images Go, HTTP handlers, panic, and deadlocks Exploring Kubernetes Operator Pattern Making Sense Out Of Cloud Native Buzz Service Discovery in Kubernetes: Combining the Best of Two Worlds API Developers Never REST How Container Networking Works: Building a Bridge Network From Scratch | iximiuz Labs Traefik: canary deployments with weighted load balancing Service Proxy, Pod, Sidecar, oh my! You Need Containers To Build Images You Don't Need an Image To Run a Container Not Every Container Has an Operating System Inside Working with container images in Go Master Go While Learning Containers Implementing Container Runtime Shim: Interactive Containers How to use Flask with gevent (uWSGI and Gunicorn editions) My 10 Years of Programming Experience Implementing Container Runtime Shim: First Code Implementing Container Runtime Shim: runc Kubernetes Repository On Flame Dealing with process termination in Linux (with Rust examples) conman - [the] Container Manager: Inception Journey From Containerization To Orchestration And Beyond Linux PTY - How docker attach and docker exec Commands Work Inside Illustrated introduction to Linux iptables From Docker Container to Bootable Linux Disk Image Пишем свой веб-сервер на Python: протокол HTTP 9001 способ создать веб-сервер на Python Explaining async/await in 200 lines of code Explaining event loop in 100 lines of code Save the day with gevent Пишем свой веб-сервер на Python: процессы, потоки и асинхронный I/O Truly optional scalar types in protobuf3 (with Go examples) Node.js Writable streams distilled Node.js Readable streams distilled How to on starting processes (mostly in Linux) Дайджест интересных ссылок – Июль 2016 Пишем свой веб-сервер на Python: сокеты Наследование в JavaScript Мастерить!
Disposable Local Development Environments with Vagrant, Docker, and Arkade
Ivan Velichko · 2021-08-03 · via Ivan on Containers, Kubernetes, and Server-Side

I use a (rather oldish) MacBook for my day-to-day development tasks. But I prefer keeping my host operating system free of development stuff. This strategy has the following benefits:

  • Increasing reproducibility of my code - it often happened to me in the past that some code worked on my machine but didn't work on others; usually, it was due to missing dependencies. Developing multiple projects on the same machine makes it harder to track what libraries and packages are required for what project. So, now I always try to have an isolated environment per project.
  • Testing code on the target platform - most of my projects have something to do with server-side and infra stuff; hence the actual target platform is Linux. Since I use a MacBook, I spend a lot of time inside virtual machines running the same operating system as my servers do. So, I'd need to duplicate the development tools from my macOS on every Linux OS I happen to use.
  • Keeping the host operating system clean and slim - even if I work on something platform-agnostic like a command-line tool, I prefer not to pollute my workstation with the dev tools and packages anyway. Projects and domains change often, and installing all the required stuff right into the host operating system would make it messy real quick.
  • Decreasing time to recover in case of machine loss - a single multi-purpose machine quickly becomes a snowflake host. Coming up with the full list of things to reinstall in the case of a sudden machine loss would be hardly feasible.

Since I usually work on several projects at the same time, I need not one but many isolated development environments. And every environment should be project-tailored, easy to spin up, suspend, and, eventually, dispose. I figured a way to achieve that by using only a few tools installed on my host operating system, and I'm going to share it here.

The approach may be helpful for folks using macOS or Linux:

  • to work on server-side and full-stack projects
  • to do Linux systems programming
  • to play with Cloud Native stack
  • to build some cool command-line tools.

Language logos

Level up your server-side game — join 20,000 engineers getting insightful learning materials straight to their inbox.

TL;DR Key takeaways

  • Use Docker for
    • running tools without installing them
    • toying with different programming languages
    • hacking playgrounds with docker-compose
  • Use Vagrant for reproducible and isolated dev environments
  • Use arkade for spinning up Kubernetes clusters in no time

What to install on the host OS

Besides everyday things like a browser or a video player, I need only the following tools installed on the host operating system:

  • Docker
  • Vagrant
  • code editors (Vim, Sublime Text, Visual Studio Code, etc.)

I deliberately keep this list short because of the reasons mentioned above. By this time, it's probably clear that the key component here is Vagrant. But I still have Docker installed on my macOS.

How Docker is used

Having Docker on the host operating systems allows me to:

  • Run arbitrary tools without installing them (e.g., alias jq='docker run -i stedolan/jq').
  • Experiment with different versions of the programming languages without setting up a full-fledged dev env.
  • Quickly hack multi-container playgrounds with docker-compose.

Docker provides fairly good isolation (assuming I trust the things I run), and it's easy to get rid of things I don't need anymore - just delete not-needed containers and images. At some point in time, I started thinking of replacing Docker with Podman. But recent Docker's promise to double down on the Development Experience gave me some extra hope for the bright future of this project.

Things I tried but abandoned in favor of the Docker + Vagrant approach:

  • homebrew - it takes forever to install something with it nowadays...
  • rvm, nvm, gvm, etc - a fine option to manage multiple versions of the same language on one machine, but requires careful use.
  • Python's venv - same as above plus a Python version manager would be needed; haven't even tried searching for it.
  • Running a dev environment inside a container - spent some time playing with this option, but IMO containers are too ephemeral to suit the use case well.

How Vagrant is used

Vagrant makes it easy to use virtual machines as development environments. I use it with the VirtualBox provider. Vagrant can be installed using homebrew or by downloading a distributive from the official website. Additionally, I always install the VirtualBox Guest Additions plugin to activate some extra functionality, such as mounting host folders into virtual machines:

$ vagrant plugin install vagrant-vbguest

When I don't know what exactly I'll need for a project, I just start with a bare minimum VM:

$ mkdir project-skunkworks
$ cd project-skunkworks

$ vagrant init debian/buster64
# vagrant init centos/7
# vagrant init alpine/alpine64
# etc

It creates a new Vagrantfile with the future virtual machine configuration. I usually adjust it then as follows:

  • Give the new virtual machine a meaningful name.
  • Configure networking connectivity with the host using a bridge adapter.
  • Mount the project's folder into the machine.
  • Adjust the allocated CPU and RAM resources.
  • Add a provisioner with the default list of packages.
  • Add a Docker or Podman provisioner (optional).

My typical initial Vagrant file looks like this:

Vagrant.configure("2") do |config|
  config.vm.box = "debian/buster64"
  config.vm.hostname = "skunkworks-01"
  config.vm.define "skunkworks-01"
  config.vagrant.plugins = ['vagrant-vbguest']

  config.vm.network "private_network", ip: "192.168.33.10"

  config.vm.synced_folder "./", "/home/vagrant/project"

  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 4
    vb.memory = "2048"
  end

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y curl git cmake tmux vim  # ...
  SHELL

  config.vm.provision "docker" do |d|
    d.run "nginx"
  end
end

Yes, it's totally fine to have Docker running inside a Vagrant virtual machine! And the Docker installation is as simple as just listing the "docker" provisioner in the file. Vagrant automagically will take care of the rest.

I try to commit the Vagrantfile with the rest of the project's files. It'd allow other developers to spin up a development environment faster. When the project's dependency list starts getting clearer, I adjust the Vagrantfile to reflect the new requirements.

To start a virtual machine using a Vagrantfile file, I run vagrant up from the folder enclosing the file. The first start may take a couple of minutes due to the initial provisioning, but the consequent starts will be significantly faster. To have shell access to the machine, I use vagrant ssh, although I need it only to run commands there. The actual code editing happens in a shared folder from the host operating system using a code editor of choice. If I run a web service in that machine, I also can access it using a host's browser on the IP address specified in the Vagrantfile.

A running virtual machine consumes a considerable amount of CPU and RAM resources, so I strive to have as few VMs running simultaneously as just possible. Using vagrant halt, I can power down the machine and power it up again later with vagrant up. A slightly more advanced use case is to suspend the machine with vagrant suspend and resume it later with vagrant resume.

A stopped virtual machine may still occupy quite some disk space, so if I don't see myself coming back to the project anytime soon, I clean up everything but the Vagrantfile with vagrant destroy.

Another useful command is vagrant global-status. Unlike other commands that were mentioned here, it doesn't need to be run from a folder containing the Vagrantfile. I use this command periodically to check which virtual machines are still alive.

Vagrantifle's are a form of Infrastructure as Code. It means that having an up-to-date Vagrantfile, I can reproduce an environment without manually reconfiguring a virtual machine. Hence, when properly used, Vagrant development environments are fully disposable.

Vagrant and Visual Studio Code remote development

When my editor/IDE of choice isn't Vim, it's either Sublime Text or Visual Studio Code. Both tools require full-fledged graphical UI support, so it becomes important to be able to run the editor on the host machine.

With a project folder mounted into a Vagrant virtual machine, it's relatively simple to point the editor to this folder and just make all the changes locally. However, more advanced projects usually require additional tools to work with the code - linters, formatters, code completion servers, etc. Installing such tools on the host machine would defeat the purpose of the isolated and disposable dev environments.

Visual Studio Code Remote Development to the rescue!

Visual Studio Code Remote Development with Vagrant Virtual Machine

Visual Studio Code Remote - SSH extension works really well with Vagrant. All I need to do is to install the extension and teach it how to access the virtual machine by providing the dump of vagrant ssh-config.

After adding the virtual machine SSH credentials, the extension will automatically install the VS Code Server on the VM. Pairing the VS Code with this server makes launching code, installing packages, and running tools remotely really seamless.

How Arkade is used

I often hack on Cloud Native stack. Or hack the Cloud Native stack. In both cases, arkade comes in handy. It's a portable binary that can download and install other command-line tools such as kubectl, kustomize, helm, kind, and many more. It also comes with tens of helm charts, so one could install a docker registry, grafana, traefik2, or even linkerd or istio service mesh into a Kubernetes cluster with just a single arkade command. Farewell, the annoying Internet search of the fresh installation instructions.

Another cool thing about it is that arkade itself can be installed with a single command:

$ curl -sLS https://get.arkade.dev | sudo sh

$ arkade --help

So, assuming I already have Docker running on my virtual machine, I can get an up and running Kubernetes cluster with an Nginx Ingress Controller in a matter of seconds!

$ arkade get kubectl
$ arkade get kind

$ kind create cluster

$ arkade install ingress-nginx

$ kubectl get all
NAME                                           READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-7d98fb5bd-956fq   1/1     Running   0          38s

NAME                                         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             LoadBalancer   10.96.239.161   <pending>     80:31973/TCP,443:31901/TCP   38s
service/ingress-nginx-controller-admission   ClusterIP      10.96.11.19     <none>        443/TCP                      38s
service/kubernetes                           ClusterIP      10.96.0.1       <none>        443/TCP                      84s
...

I find it super useful for quick experimentations. Learning about arkade broke some sort of a psychological barrier for me - now I'm less afraid of messing up with my local Kubernetes clusters because I know that I can reproduce the setup with just a few arkade get/install lines.

Combining the power of arkade with disposable vagrant virtual machines allows me to move blazingly fast on the initial phase of my projects.

Instead of conclusion

I try to keep my approach pragmatic by choosing the tools and techniques that truly work. If you read till this point, I hope there was something helpful for you in the post. If so, you may also like to read about my approach to picking a language for a programming toolbox.

Level up your server-side game — join 20,000 engineers getting insightful learning materials straight to their inbox: