The Story of How I Built a VPN protocol: Part 1
Smile
·
2026-05-02
·
via DEV Community
<h1> 🚨🚨🚨 Disclaimer 🚨🚨🚨 </h1> <p>This article and the VPN itself are written <strong>for educational purposes only</strong>.</p> <h1> How It All Started </h1> <p>I recently switched to Arch. Everything started off well: I installed all the utilities I needed, and then I decided to install the VPN I used to use. And then a problem appeared — it doesn't work on Arch (even as an AppImage).</p> <p>My provider also supported Shadowsocks, but instead of using it, I decided to write my own VPN. For more practice.</p> <h1> VPN Protocol </h1> <p>My VPN protocol is designed for maximum stealth. In my opinion, one of the most important things here is <strong>encryption from the very first packet</strong>. In my protocol, this is implemented just like in Shadowsocks — with a pre-shared key.</p> <p>Encryption algorithm: <strong>ChaCha20-Poly1305</strong>.</p> <p>It's also worth mentioning that the protocol works over <strong>TCP</strong>. A random amount of junk bytes is added to each packet for length obfuscation.</p> <h2> Packet Structure </h2> <p>Each packet has a 5-byte header that is masked as encrypted data using XOR with the first 5 bytes of the key.</p> <ul> <li> <strong>First 2 bytes</strong> — total packet length. Needed to determine where the packet ends (since TCP can segment packets).</li> <li> <strong>Third byte</strong> — flags byte. Currently only 2 flags are used: <ul> <li>Bit 1 — indicates that this packet is fake and should not be processed (not yet implemented).</li> <li>Bit 2 — flag for performing ECDH (Elliptic Curve Diffie‑Hellman).</li> </ul> </li> <li> <strong>Last 2 bytes</strong> — ciphertext length, used to separate junk bytes from the ciphertext.</li> </ul> <p>Then comes:</p> <ul> <li> <strong>12 bytes</strong> — randomly generated nonce;</li> <li> <strong>ciphertext</strong>;</li> <li> <strong>AEAD</strong> (authentication tag);</li> <li> <strong>junk bytes</strong>.</li> </ul> <h2> Handshake and Key Exchange </h2> <h3> 1. First packet from the client </h3> <p>The client sends its <strong>16-byte username</strong> to the server (encrypted, of course).</p> <h3> 2. Server response </h3> <p>If the server finds a user with that username, it:</p> <ul> <li>sends the client a randomly generated <strong>32-byte salt</strong>;</li> <li>starts computing the keys: <ul> <li> <strong>sending key</strong> (server → client)</li> <li> <strong>receiving key</strong> (server ← client)</li> </ul> </li> </ul> <h3> 3. Key computation on the server </h3> <p>The server stores the user's password in plaintext.</p> <ul> <li> <strong>Receiving key</strong> (for decrypting from the client) = hash(password + <strong>first 16 bytes</strong> of salt).</li> <li> <strong>Sending key</strong> (for encrypting to the client) = hash(password + <strong>last 16 bytes</strong> of salt).</li> </ul> <h3> 4. Client actions </h3> <p>The client receives the salt, decrypts it, and does the same thing, <strong>but the key roles are inverted</strong>:</p> <ul> <li>what is the sending key for the server becomes the receiving key for the client, and vice versa.</li> </ul> <h3> 5. ECDH and connection finalization </h3> <p>After the client has generated the keys, it generates an <strong>ephemeral key pair</strong> based on the <strong>Curve25519</strong> elliptic curve (this pair is needed for ECDH). It then sends a connection confirmation (first byte = <code>0xFF</code>) along with the public ephemeral key, setting the ECDH flag.</p> <p>The server receives the packet, deobfuscates it, and gets the confirmation and the client's ephemeral key. Then it:</p> <ul> <li>assigns an IP address to the client from a local private network;</li> <li>generates its own ephemeral key pair;</li> <li>sends the client its assigned IP address and the server's public key;</li> <li>performs the ECDH round.</li> </ul> <p>After sending, the server updates its keys by hashing the old keys with the secret obtained from ECDH.</p> <h3> 6. Client finalization </h3> <p>After receiving the packet with the IP address and the server's public ephemeral key, the client:</p> <ul> <li>creates a local tunnel;</li> <li>sets its IP address (received from the server);</li> <li>performs the ECDH round;</li> <li>updates its keys.</li> </ul> <h2> Main Work Loop </h2> <p>After the connection is established and keys are generated, the main work loop begins.</p> <h3> Client Side </h3> <p><strong>3 goroutines</strong> run on the client side:</p> <h4> First goroutine (reading from the tunnel and preparing packets) </h4> <ul> <li>Reads packets from the tunnel.</li> <li>Generates an <strong>8-byte salt</strong> to update the sending key (by hashing the old sending key with the salt).</li> <li>Adds this 8-byte salt to the beginning of the plaintext (the salt is followed by the packet read from the tunnel).</li> <li>Encrypts everything.</li> <li>Adds random junk bytes for obfuscation.</li> <li>Stores the prepared packet in a buffer.</li> </ul> <h4> Second goroutine (sending packets) </h4> <ul> <li>Responsible for sending already prepared packets.</li> <li>Packets are sent in <strong>batches of 1 to 5 packets</strong> (the protocol is of course segmented at OSI layers 3 and 4, but I can't influence that).</li> </ul> <h4> Third goroutine (receiving packets from the server) </h4> <ul> <li>Responsible for receiving packets from the server.</li> <li>Performs deobfuscation and decryption.</li> <li>Writes the decrypted data to the tunnel.</li> </ul> <h3> Server Side </h3> <p>The server has <strong>3 main goroutines</strong>, plus additional goroutines for receiving packets from clients.</p> <h4> First goroutine (handshake handling) </h4> <p>Handles incoming handshake requests from clients. If the handshake is successful, a <strong>new goroutine</strong> is created to process packets sent by that client.</p> <h4> Second goroutine (reading from the tunnel) </h4> <p>Reads packets from the tunnel and sends them to clients.</p> <h4> Third goroutine (cleaning inactive connections) </h4> <p>Cleans up inactive connections.</p> <h2> Key Updates </h2> <h3> Salt in every packet </h3> <p>Every packet (whether from client or server) contains a <strong>salt</strong>. It is used to update the keys:</p> <ul> <li> <strong>The server</strong>, when sending a packet, includes a salt. After sending, it updates its <strong>sending key</strong> by hashing the old key with that salt.</li> <li> <strong>The client</strong>, when receiving and decrypting a packet, also updates a key — but not the sending key, the <strong>receiving key</strong>.</li> <li>When the client sends a packet, the same happens, but the roles are reversed.</li> </ul> <h3> Periodic ECDH updates </h3> <p>Every <strong>4 minutes</strong> or after sending <strong>2³² packets</strong> (whichever comes first), keys are updated using ECDH on elliptic curves. The keys are transmitted along with data packets.</p> <p>And that, in fact, is the entire protocol. During implementation, I thought about writing it in <strong>Go</strong> or <strong>Rust</strong>. I chose Go for its simplicity.</p> <h1> Implementation Process </h1> <p>To be honest, the protocol architecture was mostly developed while writing the code. It has quite a few problems — both in terms of protocol design and implementation.</p> <h3> Example problems </h3> <ul> <li><p><strong>Constant username packet length</strong><br><br> The encrypted username packet has a constant length of 44 bytes (12 bytes nonce, 16 bytes ciphertext, and a 16-byte AEAD tag). Knowing this and that the user is using this protocol, you can calculate the 4th and 5th bytes of the key.</p></li> <li><p><strong>Repository duplication</strong><br><br> I foolishly created two separate repositories — one for the client and one for the server. As a result, the branches containing common modules just duplicate each other.</p></li> <li><p><strong>Git flow</strong><br><br> I tried to follow git flow, but failed here too.</p></li> <li><p><strong>Vulnerabilities</strong><br><br> I also have a feeling that there are more vulnerabilities in the code than working logic.</p></li> <li><p><strong>No graceful shutdown</strong><br><br> There is no proper negotiated client-server disconnect — just a connection break.</p></li> </ul> <p>Although considering this is my first project, I think it didn't turn out too badly. If anyone wants to check out this mess, here are the links:</p> <ul> <li> <strong>Client</strong>: <a href="https://github.com/SmileUwUI/smileTun-client" rel="noopener noreferrer">https://github.com/SmileUwUI/smileTun-client</a> </li> <li> <strong>Server</strong>: <a href="https://github.com/SmileUwUI/smileTun-server" rel="noopener noreferrer">https://github.com/SmileUwUI/smileTun-server</a> </li> </ul> <p>Currently, the implementation works. And I'm writing this article through my own VPN protocol.</p> <h1> Future Plans </h1> <ul> <li>Merge both repositories into one.</li> <li>Add fake packet sending.</li> <li>Add TLS mimicry.</li> <li>And much more.</li> </ul> <p>If anyone has any questions or recommendations — leave them in the comments. For now, I bid you farewell. Good luck to everyone!</p>
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。