Eli Bendersky's website - Network Programminghttps://eli.thegreenplace.net/2024-03-04T13:22:43-08:00Sign in with Google in Go2024-01-13T06:33:00-08:002024-01-13T14:39:52-08:00Eli Benderskytag:eli.thegreenplace.net,2024-01-13:/2024/sign-in-with-google-in-go/<p>This post provides some code samples for implementing a "Sign-in with Google"
flow for your web application in Go. For an overview of auth/authz and the OAuth
protocol, please refer to my earlier post about
<a class="reference external" href="https://eli.thegreenplace.net/2023/sign-in-with-github-in-go/">Sign-in with GitHub</a>.</p>
<p>Sign-in with Google has existed in one form or another for …</p><p>This post provides some code samples for implementing a "Sign-in with Google"
flow for your web application in Go. For an overview of auth/authz and the OAuth
protocol, please refer to my earlier post about
<a class="reference external" href="https://eli.thegreenplace.net/2023/sign-in-with-github-in-go/">Sign-in with GitHub</a>.</p>
<p>Sign-in with Google has existed in one form or another for many years, and
the technical approach to it evolved over time. I will start by presenting
the currently recommended way - and the one that will feel most familiar to
users - and will then mention a slightly more complicated and flexible
alternative.</p>
<div class="section" id="using-google-identity-service-gis">
<h2>Using Google Identity Service (GIS)</h2>
<p>The currently recommended way to implement sign-in with Google is using the
<a class="reference external" href="https://developers.google.com/identity/gsi/web/guides/overview">Google Identity Service client library</a>. There's a
lot of documentation on that page, and it's worth reading through.</p>
<p>Using this approach, we'll get a standard looking button:</p>
<img alt="Sample screen of sign-in with Google" class="align-center" src="https://eli.thegreenplace.net/images/2024/gis-signin-google.png" />
<p>When clicked, a popup opens with a standard-looking Google account selection:</p>
<img alt="The popup window that opens for Google sign-in" class="align-center" src="https://eli.thegreenplace.net/images/2024/google-signin-popup.png" />
<p>There's also an option for "one tap" which surfaces the currently logged in
Google user in an in-page popup to click (this also works well on mobile!)</p>
<p>To get started, you'll need to register an application at
<a class="reference external" href="https://console.cloud.google.com/apis/credentials">https://console.cloud.google.com/apis/credentials</a> and obtain a client ID and
secret (the secret isn't used for this approach, but is needed for the next
one).</p>
<p>We'll use a Google-provided JS client library called GSI which we include in
our web page; it implements the actual button and all the client-side handling;
here's a snippet <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2024/go-google-login/gsi-idtoken">from our Go server for this sample</a>:</p>
<div class="highlight"><pre><span></span><span class="c1">// This should be taken from https://console.cloud.google.com/apis/credentials</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">GoogleClientID</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Getenv</span><span class="p">(</span><span class="s">"GOOGLE_CLIENT_ID"</span><span class="p">)</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">servingSchema</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"http://"</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">servingAddress</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"localhost:8080"</span><span class="w"></span>
<span class="kd">const</span><span class="w"> </span><span class="nx">callbackPath</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">"/google/callback"</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">rootHtmlTemplate</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">template</span><span class="p">.</span><span class="nx">Must</span><span class="p">(</span><span class="nx">template</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="s">"root"</span><span class="p">).</span><span class="nx">Parse</span><span class="p">(</span><span class="s">`</span>
<span class="s"><!DOCTYPE html></span>
<span class="s"><html></span>
<span class="s"><body></span>
<span class="s"> <script src="https://accounts.google.com/gsi/client" async></script></span>
<span class="s"> <h1>Welcome to this web app!</h1></span>
<span class="s"> <p>Let's sign in with Google:</p></span>
<span class="s"> <div</span>
<span class="s"> id="g_id_onload"</span>
<span class="s"> data-client_id="{{.ClientID}}"</span>
<span class="s"> data-login_uri="{{.CallbackUrl}}"></span>
<span class="s"> </div></span>
<span class="s"> <div</span>
<span class="s"> class="g_id_signin"</span>
<span class="s"> data-type="standard"</span>
<span class="s"> data-theme="filled_blue"</span>
<span class="s"> data-text="sign_in_with"</span>
<span class="s"> data-shape="rectangular"</span>
<span class="s"> data-width="200"</span>
<span class="s"> data-logo_alignment="left"></span>
<span class="s"> </div></span>
<span class="s"></body></span>
<span class="s"></html></span>
<span class="s">`</span><span class="p">))</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">GoogleClientID</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Set GOOGLE_CLIENT_ID env var"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span><span class="w"> </span><span class="nx">rootHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="nx">callbackPath</span><span class="p">,</span><span class="w"> </span><span class="nx">callbackHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"Listening on: %s%s\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">servingSchema</span><span class="p">,</span><span class="w"> </span><span class="nx">servingAddress</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Panic</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">servingAddress</span><span class="p">,</span><span class="w"> </span><span class="nx">mux</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">rootHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">rootHtmlTemplate</span><span class="p">.</span><span class="nx">Execute</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"CallbackUrl"</span><span class="p">:</span><span class="w"> </span><span class="nx">servingSchema</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">servingAddress</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">callbackPath</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"ClientID"</span><span class="p">:</span><span class="w"> </span><span class="nx">GoogleClientID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Note the configuration on the <tt class="docutils literal"><div></tt> element for the button; there's
a ton of potential customization there - <a class="reference external" href="https://developers.google.com/identity/gsi/web/reference/html-reference">the docs</a>
explain these in detail.</p>
<p>We serve this page on the root (<tt class="docutils literal">"/"</tt>) handler of our local web server, and
also register a callback URL which the Google auth server will redirect to
once it confirms the user's identity <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.
When it does, it sends the logged-in user's information as a JSON Web Token
(JWT); our server-side application should
<a class="reference external" href="https://developers.google.com/identity/gsi/web/guides/verify-google-id-token">verify the validity of the token</a>
and then it's ready to accept the signed-in user.</p>
<p>We'll be using the <tt class="docutils literal">google.golang.org/api/idtoken</tt> package for token
verification (this comes from the official GCP client library for Go); here's
our callback:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">callbackHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">ParseForm</span><span class="p">();</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The following steps follow</span><span class="w"></span>
<span class="w"> </span><span class="c1">// https://developers.google.com/identity/gsi/web/guides/verify-google-id-token</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 1. Verify the CSRF token, which uses the double-submit-cookie pattern and</span><span class="w"></span>
<span class="w"> </span><span class="c1">// is added both as a cookie value and post body.</span><span class="w"></span>
<span class="w"> </span><span class="nx">token</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Cookie</span><span class="p">(</span><span class="s">"g_csrf_token"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"token not found"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">bodyToken</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">FormValue</span><span class="p">(</span><span class="s">"g_csrf_token"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">token</span><span class="p">.</span><span class="nx">Value</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">bodyToken</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"token mismatch"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 2. Verify the ID token, which is returned in the `credential` field.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We use the idtoken package for this. `audience` is our client ID.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">validator</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">idtoken</span><span class="p">.</span><span class="nx">NewValidator</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">credential</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">FormValue</span><span class="p">(</span><span class="s">"credential"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">payload</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">validator</span><span class="p">.</span><span class="nx">Validate</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span><span class="w"> </span><span class="nx">credential</span><span class="p">,</span><span class="w"> </span><span class="nx">GoogleClientID</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 3. Once the token's validity is confirmed, we can use the user identifying</span><span class="w"></span>
<span class="w"> </span><span class="c1">// information in the Google ID token.</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nx">k</span><span class="p">,</span><span class="w"> </span><span class="nx">v</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">payload</span><span class="p">.</span><span class="nx">Claims</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"%v: %v\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">k</span><span class="p">,</span><span class="w"> </span><span class="nx">v</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The numbered comments follow the documentation step by step, so it should be
easy to decipher. The end result is dumping some information about the user
from the token; specifically, the Google email and registered name will be
provided. Once you have that, you know who the user is and that they have an
actual Google account.</p>
<p>This is it! I like how all the complicated front-end details are handled
by Google's own JS library; there's a lot of potential nuance there with
one-tap and automatic sign-in, different UI requirements for desktop browsers
and mobile, etc. If you're developing a standard web-app, this definitely
seems like the way to go.</p>
</div>
<div class="section" id="using-openid-connect">
<h2>Using OpenID Connect</h2>
<p><a class="reference external" href="https://en.wikipedia.org/wiki/OpenID">OpenID Connect</a> is an authentication
protocol built on top of OAuth 2. The Google documentation for integrating it
into your applications <a class="reference external" href="https://developers.google.com/identity/openid-connect/openid-connect">is here</a>.</p>
<p>This is a more generic approach than using GIS - you're not beholden to use
specific JS or HTML elements, but can integrate it into your flow in whatever
way you wish - similarly to the <a class="reference external" href="https://eli.thegreenplace.net/2023/sign-in-with-github-in-go/">GitHub approach outlined in my last post</a>.
I won't paste the code here - but it's well documented and follows the official
docs. I have two separate samples:</p>
<ol class="arabic simple">
<li>Using the <a class="reference external" href="https://pkg.go.dev/github.com/coreos/go-oidc/v3/oidc">go-oidc package</a>:
the <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2024/go-google-login/oidc">full code is here</a>.</li>
<li>Using the <a class="reference external" href="https://pkg.go.dev/github.com/dghubble/gologin/v2">gologin package</a>:
the <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2024/go-google-login/gologin">full code is here</a>.
You may recognize the <tt class="docutils literal">gologin</tt> package from my previous post on GitHub
sign-in; indeed, the code delta between the two options is very small!</li>
</ol>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Please refer to my <a class="reference external" href="https://eli.thegreenplace.net/2023/sign-in-with-github-in-go/">description of OAuth</a>
to understand the redirection flow. This Google sign-in flow still relies
on OAuth underneath.</td></tr>
</tbody>
</table>
</div>
Sign in with GitHub in Go2023-12-09T14:15:00-08:002024-01-13T14:39:52-08:00Eli Benderskytag:eli.thegreenplace.net,2023-12-09:/2023/sign-in-with-github-in-go/<p>It's common to see web applications that let you log in through third-party
services. "Sign in with Google" is particularly popular; on developer-oriented
websites, "Sign in with GitHub" also pops up quite a bit. In this post, I want
to briefly explore OAuth - the technology that enables these delegated logins …</p><p>It's common to see web applications that let you log in through third-party
services. "Sign in with Google" is particularly popular; on developer-oriented
websites, "Sign in with GitHub" also pops up quite a bit. In this post, I want
to briefly explore OAuth - the technology that enables these delegated logins,
and present some ways to integrate GitHub login in your Go service.
<a class="reference external" href="https://eli.thegreenplace.net/2024/sign-in-with-google-in-go/">Signing in through Google is covered in a separate post</a>.</p>
<p>A note about authentication terminology:</p>
<ul class="simple">
<li>Authentication (authn): is the process of verifying the identity of a user
or entity. It answers the question, "Who are you?", typically
through credentials like usernames and passwords, 2FA etc.</li>
<li>Authorization (authz): is the process of determining what permissions an
authenticated user has with a given service (e.g. Editor, Commenter or
Viewer on Google Documents).</li>
</ul>
<p>This post is about authn, though GitHub really provides a more general authz
mechanism. In GitHub, when you attempt to use OAuth login, you ask for specific
permissions (called "scopes") ahead of time; so a user authentication process
combines authn (does this user have a valid GitHub account?) with authz
(can this app get the following permissions to the user's account?)</p>
<p>We try to focus only on authn though, not asking GitHub for any particular
permissions other than verifying that a user has an account and getting some
basic user information (email) that can be used to uniquely identify the user
in our application.</p>
<p>This post provides code samples for accomplishing this task in three ways:</p>
<ol class="arabic simple">
<li>Using nothing but the Go standard library</li>
<li>Using a semi-standard package for handling some of the OAuth minutiae,
saving some code</li>
<li>Using a third-party package for handling more of the process, saving
even more code</li>
</ol>
<div class="section" id="a-brief-overview-of-oauth">
<h2>A brief overview of OAuth</h2>
<p>OAuth is an authorization standard; currently in version 2.0, it's formally
described in <a class="reference external" href="https://datatracker.ietf.org/doc/html/rfc6750">RFC 6750</a>.
In this post I'll only provide a brief introduction to the standard in the
context of the GitHub authentication flow I'm describing; I recommend reading
more - start with the <a class="reference external" href="https://en.wikipedia.org/wiki/OAuth">Wikipedia page</a>
and follow links from there as needed.</p>
<p>Here's a useful diagram describing the OAuth 2.0 (it's taken from
<a class="reference external" href="https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2">this post by Digital Ocean</a>,
which is also a good introduction to the subject):</p>
<img alt="Diagram describing the steps and actors in an OAuth 2 auth flow" class="align-center" src="https://eli.thegreenplace.net/images/2023/auth-flow-digitalocean.png" />
<p>When we want to provide a "Sign-in with GitHub" in our application, the actors
involved in the diagram are:</p>
<ul class="simple">
<li>Application (or client): is our application - a web app that wants to allow
users to log in with their GitHub account rather than (or in addition to)
implementing its own authentication flow.</li>
<li>User: wants to log into our app, and has a GitHub account.</li>
<li>User-agent: the user's web browser</li>
<li>Auth Server: the GitHub OAuth authorization server</li>
</ul>
<p>The steps involved are:</p>
<ol class="arabic simple">
<li>The user visits the application's website and chooses to log in with GitHub
credentials. The application redirects the user to the GitHub auth server
which notifies them that "Application FOO" wants them to log in.</li>
<li>The user logs into GitHub; note that this is happening on GitHub's website,
in a secured HTTPS session vs. GitHub's server. The user enters their
email, password and 2FA if required.</li>
<li>If the login is successful, the auth server redirects the user's browser
back to the application, and provides the application with a temporary code
it can use to ask for access tokens.</li>
<li>The application uses the access token from GitHub along with a secret it
has pre-registered with GitHub when it was created to request an access
token to the user's account.</li>
<li>GitHub checks that everything is kosher and provides an access token to
the application.</li>
</ol>
<p>From this stage on, the application can use the token to query the GitHub API
on behalf of the logged-in user, based on the permissions (or "scopes") the
user agreed to provide to the application <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>. For logging in, the application
just needs to know what the user's email address is.</p>
<p>One important aspect to understand about OAuth for web applications is that
it leverages HTTP redirection (status code 301) to route the user between
the different servers involved. This is important for user control, security
and also convenience (for example, it allows us to register <tt class="docutils literal">localhost</tt>
endpoints with GitHub for testing). The way it works is that in step (1) when
our application sends the user to GitHub, it adds a redirection URL
for GitHub in a request parameter; GitHub uses this to redirect the user back
to our page in step (3). We'll see this in action soon.</p>
</div>
<div class="section" id="sample-1-github-oauth-flow-using-raw-stdlib">
<h2>Sample 1: GitHub OAuth flow using raw stdlib</h2>
<p>In <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-github-login/using-stdlib-only">our first sample</a>,
we're not going to be using any 3rd party
packages; instead, we implement a complete GitHub auth flow using the Go
standard library.</p>
<p>This sample follows the GitHub documentation: <a class="reference external" href="https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps">Authorizing OAuth apps</a>,
as described in the "Web application" flow <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>. The steps outlined on that page
are directly mapped to steps in my sample Go program. If you want to follow
along, make sure to <a class="reference external" href="https://github.com/settings/applications/new">register an application with GitHub</a>,
set a compatible callback path and write down your client ID and client secret;
my code expects these to be in the env vars <tt class="docutils literal">GITHUB_CLIENT_ID</tt> and
<tt class="docutils literal">GITHUB_CLIENT_SECRET</tt>, respectively.</p>
<p>Our application starts by registering some HTTP routes:</p>
<div class="highlight"><pre><span></span><span class="c1">// These should be taken from your GitHub application settings</span><span class="w"></span>
<span class="c1">// at https://github.com/settings/developers</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">GithubClientID</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Getenv</span><span class="p">(</span><span class="s">"GITHUB_CLIENT_ID"</span><span class="p">)</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">GithubClientSecret</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Getenv</span><span class="p">(</span><span class="s">"GITHUB_CLIENT_SECRET"</span><span class="p">)</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">GithubClientID</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">GithubClientSecret</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Set GITHUB_CLIENT_* env vars"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span><span class="w"> </span><span class="nx">rootHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/login/"</span><span class="p">,</span><span class="w"> </span><span class="nx">githubLoginHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/github/callback/"</span><span class="p">,</span><span class="w"> </span><span class="nx">githubCallbackHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">"localhost:8080"</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"Listening on: http://%s\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">addr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Panic</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>When the server runs, the user is expected to visit the root route <tt class="docutils literal">/</tt>, where
they're presented with a link to "Log in with GitHub". This is done with the
following handler:</p>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">rootHTML</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">`</span>
<span class="s"><h1>My web app</h1></span>
<span class="s"><p>Using raw HTTP OAuth 2.0</p></span>
<span class="s"><p>You can log into this app with your GitHub credentials:</p></span>
<span class="s"><p><a href="/login/">Log in with GitHub</a></p></span>
<span class="s">`</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">rootHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">rootHTML</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Once the user clicks the "log in" link, they're taken to the <tt class="docutils literal">/login/</tt>
handler:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">githubLoginHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Step 1: Request a user's GitHub identity</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ... by redirecting the user's browser to a GitHub login endpoint. We're not</span><span class="w"></span>
<span class="w"> </span><span class="c1">// setting redirect_uri, leaving it to GitHub to use the default we set for</span><span class="w"></span>
<span class="w"> </span><span class="c1">// this application: /github/callback</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We're also not asking for any specific scope, because we only need access</span><span class="w"></span>
<span class="w"> </span><span class="c1">// to the user's public information to know that the user is really logged in.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We're setting a random state cookie for the client to return</span><span class="w"></span>
<span class="w"> </span><span class="c1">// to us when the call comes back, to prevent CSRF per</span><span class="w"></span>
<span class="w"> </span><span class="c1">// section 10.12 of https://www.rfc-editor.org/rfc/rfc6749.html</span><span class="w"></span>
<span class="w"> </span><span class="nx">state</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">randString</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">c</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">http</span><span class="p">.</span><span class="nx">Cookie</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Name</span><span class="p">:</span><span class="w"> </span><span class="s">"state"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="nx">state</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Path</span><span class="p">:</span><span class="w"> </span><span class="s">"/"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">MaxAge</span><span class="p">:</span><span class="w"> </span><span class="nb">int</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">.</span><span class="nx">Seconds</span><span class="p">()),</span><span class="w"></span>
<span class="w"> </span><span class="nx">Secure</span><span class="p">:</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">TLS</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HttpOnly</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">SetCookie</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">c</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">redirectURL</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"https://github.com/login/oauth/authorize?client_id=%s&state=%s"</span><span class="p">,</span><span class="w"> </span><span class="nx">GithubClientID</span><span class="p">,</span><span class="w"> </span><span class="nx">state</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Redirect</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">,</span><span class="w"> </span><span class="nx">redirectURL</span><span class="p">,</span><span class="w"> </span><span class="mi">301</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The comment explains how this maps to step 1 in the <a class="reference external" href="https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps">documented GitHub flow</a>.
The note about <tt class="docutils literal">redirect_url</tt> is important; typically the auth
server expects a <tt class="docutils literal">redirect_url</tt> path. However, it should also be configured
when you register the application. Here I opt for using the default
configuration and am not passing <tt class="docutils literal">redirect_url</tt> explicitly.</p>
<p>This code implements CSRF protection by generating a random string, storing
it in a cookie for this user's session and sending it with the <tt class="docutils literal">state</tt> URL
parameter to GitHub. When GitHub calls our redirect URL, it will attach this
state - we can thus verify the request is legit and not a malicious attack.</p>
<p>At this point the user is presented with a familiar "sign-in to GitHub" screen
on GitHub's website:</p>
<img alt="Sign-in screenshot from GitHub" class="align-center" src="https://eli.thegreenplace.net/images/2023/github-auth-login.png" />
<p>The user verifies their credentials vs. GitHub; then GitHub redirects the user's
browser back to the redirect URL provided (or configured in the GitHub app
settings). In our case, the redirect goes to
<a class="reference external" href="http://localhost:8080/github/callback/">http://localhost:8080/github/callback/</a> <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a>, which is mapped to
<tt class="docutils literal">githubCallbackHandler</tt>. Here's the first part of this function, implementing
step 2 from the documented GitHub flow:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">githubCallbackHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Step 2: Users are redirected back to your site by GitHub</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// The user is authenticated w/ GitHub by this point, and GH provides us</span><span class="w"></span>
<span class="w"> </span><span class="c1">// a temporary code we can exchange for an access token using the app's</span><span class="w"></span>
<span class="w"> </span><span class="c1">// full credentials.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Start by checking the state returned by GitHub matches what</span><span class="w"></span>
<span class="w"> </span><span class="c1">// we've stored in the cookie.</span><span class="w"></span>
<span class="w"> </span><span class="nx">state</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Cookie</span><span class="p">(</span><span class="s">"state"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"state not found"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Query</span><span class="p">().</span><span class="nx">Get</span><span class="p">(</span><span class="s">"state"</span><span class="p">)</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"state did not match"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We use the code, alongside our client ID and secret to ask GH for an</span><span class="w"></span>
<span class="w"> </span><span class="c1">// access token to the API.</span><span class="w"></span>
<span class="w"> </span><span class="nx">code</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Query</span><span class="p">().</span><span class="nx">Get</span><span class="p">(</span><span class="s">"code"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">requestBodyMap</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"client_id"</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"client_secret"</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientSecret</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"code"</span><span class="p">:</span><span class="w"> </span><span class="nx">code</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">requestJSON</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">requestBodyMap</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">req</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewRequest</span><span class="p">(</span><span class="s">"POST"</span><span class="p">,</span><span class="w"> </span><span class="s">"https://github.com/login/oauth/access_token"</span><span class="p">,</span><span class="w"> </span><span class="nx">bytes</span><span class="p">.</span><span class="nx">NewBuffer</span><span class="p">(</span><span class="nx">requestJSON</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Accept"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">DefaultClient</span><span class="p">.</span><span class="nx">Do</span><span class="p">(</span><span class="nx">req</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"unable to connect to access_token endpoint"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusInternalServerError</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">respbody</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Represents the response received from Github</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">ghresp</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">AccessToken</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`json:"access_token"`</span><span class="w"></span>
<span class="w"> </span><span class="nx">TokenType</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`json:"token_type"`</span><span class="w"></span>
<span class="w"> </span><span class="nx">Scope</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="s">`json:"scope"`</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Unmarshal</span><span class="p">(</span><span class="nx">respbody</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">ghresp</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// {...}</span><span class="w"></span>
</pre></div>
<p>The function does just what it says on the box: after verifying the CSRF token,
it provides the code it got from GitHub back to GitHub's OAuth access token
endpoint, along with a secret shared only by the application and GitHub
(it's in the app settings). In return it gets a token that can be used as
a bearer token for HTTP auth when accessing the GitHub API on behalf of the
logged-in user. Here's the rest of the code:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">githubCallbackHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// {...}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Step 3: Use the access token to access the API</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// With the access token in hand, we can access the GitHub API on behalf</span><span class="w"></span>
<span class="w"> </span><span class="c1">// of the user. Since we didn't provide a scope, we only get access to</span><span class="w"></span>
<span class="w"> </span><span class="c1">// the user's public information.</span><span class="w"></span>
<span class="w"> </span><span class="nx">userInfo</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">getGitHubUserInfo</span><span class="p">(</span><span class="nx">ghresp</span><span class="p">.</span><span class="nx">AccessToken</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-type"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">userInfo</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// getGitHubUserInfo queries GitHub's user API for information about the</span><span class="w"></span>
<span class="c1">// authorized user, given the access token received earlier.</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">getGitHubUserInfo</span><span class="p">(</span><span class="nx">accessToken</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Query the GH API for user info</span><span class="w"></span>
<span class="w"> </span><span class="nx">req</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewRequest</span><span class="p">(</span><span class="s">"GET"</span><span class="p">,</span><span class="w"> </span><span class="s">"https://api.github.com/user"</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Authorization"</span><span class="p">,</span><span class="w"> </span><span class="s">"Bearer "</span><span class="o">+</span><span class="nx">accessToken</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">DefaultClient</span><span class="p">.</span><span class="nx">Do</span><span class="p">(</span><span class="nx">req</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">respbody</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">respbody</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The end result of this is dumping a JSON map with the logged-in user's
information to output. This is just an example, of course. A real web
application would now use the user's email as a unique ID to look up
things in its own DB - it knows the user is authenticated now!</p>
</div>
<div class="section" id="sample-2-using-the-x-oauth2-package">
<h2>Sample 2: using the x/oauth2 package</h2>
<p><a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-github-login/using-x-oauth2-package">The second sample</a>
uses the <a class="reference external" href="https://pkg.go.dev/golang.org/x/oauth2">golang.org/x/oauth2 package</a>
to take care of some of the OAuth
details instead of doing everything manually. Much of the code remains the same
- I just want to highlight some places where <tt class="docutils literal">x/oauth2</tt> is used instead of
writing code from scratch. First, we create a configuration struct in our
<tt class="docutils literal">main</tt> function:</p>
<div class="highlight"><pre><span></span><span class="nx">conf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">oauth2</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ClientID</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">ClientSecret</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientSecret</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Scopes</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="nx">Endpoint</span><span class="p">:</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">Endpoint</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Here <tt class="docutils literal">github</tt> refers to the <a class="reference external" href="https://pkg.go.dev/golang.org/x/oauth2@v0.15.0/github">x/oauth2/github subpackage</a>, which
holds some constants useful for GitHub OAuth. Specifically, <tt class="docutils literal">github.Endpoint</tt>
is an alias to:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">GitHub</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">oauth2</span><span class="p">.</span><span class="nx">Endpoint</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">AuthURL</span><span class="p">:</span><span class="w"> </span><span class="s">"https://github.com/login/oauth/authorize"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">TokenURL</span><span class="p">:</span><span class="w"> </span><span class="s">"https://github.com/login/oauth/access_token"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">DeviceAuthURL</span><span class="p">:</span><span class="w"> </span><span class="s">"https://github.com/login/device/code"</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>So our code needn't have these paths hard-coded.</p>
<p>In <tt class="docutils literal">githubLoginHandler</tt>, we can use the <a class="reference external" href="https://pkg.go.dev/golang.org/x/oauth2#Config.AuthCodeURL">AuthCodeURL method</a> instead of
constructing the URL manually.</p>
<p>Finally, in <tt class="docutils literal">githubCallbackHandler</tt> we get help in two ways:</p>
<div class="highlight"><pre><span></span><span class="nx">code</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Query</span><span class="p">().</span><span class="nx">Get</span><span class="p">(</span><span class="s">"code"</span><span class="p">)</span><span class="w"></span>
<span class="nx">tok</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">lf</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Exchange</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">code</span><span class="p">)</span><span class="w"> </span><span class="c1">// 1: token exchange</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// This client will have a bearer token to access the GitHub API on</span><span class="w"></span>
<span class="c1">// the user's behalf.</span><span class="w"></span>
<span class="nx">client</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">lf</span><span class="p">.</span><span class="nx">conf</span><span class="p">.</span><span class="nx">Client</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nx">Background</span><span class="p">(),</span><span class="w"> </span><span class="nx">tok</span><span class="p">)</span><span class="w"> </span><span class="c1">// 2: client with token</span><span class="w"></span>
<span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"https://api.github.com/user"</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="nx">respbody</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w"></span>
<span class="nx">userInfo</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">respbody</span><span class="p">)</span><span class="w"></span>
<span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-type"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">userInfo</span><span class="p">))</span><span class="w"></span>
</pre></div>
<p>First, we can call <tt class="docutils literal">Exchange</tt> to ask <tt class="docutils literal">x/oauth2</tt> to perform the "code for
token" exchange step. Second, we get an HTTP client with the bearer token
already configured and don't have to do it manually. The resulting code is
somewhat shorter and has fewer hard-coded operations, delegating some of the
work to the <tt class="docutils literal">x/oauth2</tt> package.</p>
</div>
<div class="section" id="sample-3-using-the-gologin-package">
<h2>Sample 3: using the gologin package</h2>
<p>Finally, <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-github-login/using-gologin">our third sample</a>
uses the third-party
<a class="reference external" href="https://pkg.go.dev/github.com/dghubble/gologin">gologin package</a>.
<tt class="docutils literal">gologin</tt> implements helpers for logging into various services like GitHub,
Google and Twitter. It encapsulates even more functionality. Here's the
relevant part of the <tt class="docutils literal">main</tt> function in this sample:</p>
<div class="highlight"><pre><span></span><span class="nx">conf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">oauth2</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ClientID</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">ClientSecret</span><span class="p">:</span><span class="w"> </span><span class="nx">GithubClientSecret</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Scopes</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="nx">Endpoint</span><span class="p">:</span><span class="w"> </span><span class="nx">oauth2github</span><span class="p">.</span><span class="nx">Endpoint</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// gologin has a default cookie configuration for debug deployments (no TLS).</span><span class="w"></span>
<span class="nx">cookieConf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">gologin</span><span class="p">.</span><span class="nx">DebugOnlyCookieConfig</span><span class="w"></span>
<span class="nx">http</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span><span class="w"> </span><span class="nx">rootHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">loginHandler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">LoginHandler</span><span class="p">(</span><span class="nx">conf</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w"></span>
<span class="nx">http</span><span class="p">.</span><span class="nx">Handle</span><span class="p">(</span><span class="s">"/login/"</span><span class="p">,</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">StateHandler</span><span class="p">(</span><span class="nx">cookieConf</span><span class="p">,</span><span class="w"> </span><span class="nx">loginHandler</span><span class="p">))</span><span class="w"></span>
<span class="nx">callbackHandler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">CallbackHandler</span><span class="p">(</span><span class="nx">conf</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandlerFunc</span><span class="p">(</span><span class="nx">githubCallbackHandler</span><span class="p">),</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w"></span>
<span class="nx">http</span><span class="p">.</span><span class="nx">Handle</span><span class="p">(</span><span class="nx">callbackPath</span><span class="p">,</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">StateHandler</span><span class="p">(</span><span class="nx">cookieConf</span><span class="p">,</span><span class="w"> </span><span class="nx">callbackHandler</span><span class="p">))</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">gologin</tt> relies upon the <tt class="docutils literal">x/oauth2</tt> package and uses its <tt class="docutils literal">Config</tt>. The
rest of the code registers the HTTP handlers, and it has two interesting
properties:</p>
<ol class="arabic simple">
<li><tt class="docutils literal">gologin</tt> handles the CSRF cookie management for us, using its
<tt class="docutils literal">StateHandler</tt> middleware.</li>
<li>It also handles the token exchange with GitHub automatically, by exposing
a <tt class="docutils literal">CallbackHandler</tt> middleware that wraps our <tt class="docutils literal">githubCallbackHandler</tt>
handler.</li>
</ol>
<p>Our handler gets user information through the request context:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">githubCallbackHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ctx</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Context</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">githubUser</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">github</span><span class="p">.</span><span class="nx">UserFromContext</span><span class="p">(</span><span class="nx">ctx</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusInternalServerError</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-type"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">buf</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">json</span><span class="p">.</span><span class="nx">Marshal</span><span class="p">(</span><span class="nx">githubUser</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">buf</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>That's about it! <tt class="docutils literal">gologin</tt> encapsulates the rest of the functionality through
its middleware, enabling GitHub logins in only a handful of lines of code.</p>
</div>
<div class="section" id="which-approach-to-choose">
<h2>Which approach to choose?</h2>
<p>All the samples in this post accomplish the same goal - they let a web app
delegate its user authentication to GitHub. The samples are presented in
increasing level of encapsulation and external dependency - from a completely
raw HTTP-based approach using only the stdlib, to using a package that does
almost all the work for us.</p>
<p>How to choose between these is a question every project may answer differently,
based on their stage, resources and <a class="reference external" href="https://eli.thegreenplace.net/2017/benefits-of-dependencies-in-software-projects-as-a-function-of-effort/">approach to dependencies</a>.
I'd probably go for the second approach - using <tt class="docutils literal">x/oauth2</tt> - since it balances
dependencies with utility.</p>
<p>The <tt class="docutils literal">x/oauth2</tt> package is "semi-official", maintained by some members of Go
team and other Go users at Google, and it's helpful without being magical. That
said, the <tt class="docutils literal">gologin</tt> package also looks pretty solid, so that could be a good
option too if your tolerance to third-party dependencies is higher.</p>
</div>
<div class="section" id="code">
<h2>Code</h2>
<p>The full code for this post is available <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-github-login">on GitHub</a>.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>This is an artifact of using GitHub's authz flow for authn. Generally,
this flow allows applications (think CI systems) to have broader access
to the user's GitHub account in order to render some useful service
to the user. In our case we only use the flow for authn, so all we need
is "basic account access", to know that the user indeed has valid access
to a GitHub account and the public details of this account.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Kudos to GitHub for the solid documentation they have on this topic.
They also have a complete <a class="reference external" href="https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/building-a-login-with-github-button-with-a-github-app">sample in Ruby</a>
which is similar to my Sample 1.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>This is a good time to admire the choice OAuth made to use redirects for
this step. Consider the following questions: without redirects, how
would you tell GitHub to dial to some arbitrary path on your <tt class="docutils literal">localhost</tt>?
Also, without redirects how could this request use the user's browser
cookies to enable easy storage of CSRF tokens?</td></tr>
</tbody>
</table>
</div>
Better HTTP server routing in Go 1.222023-10-16T05:17:00-07:002023-10-16T12:26:41-07:00Eli Benderskytag:eli.thegreenplace.net,2023-10-16:/2023/better-http-server-routing-in-go-122/<p>An <a class="reference external" href="https://github.com/golang/go/issues/61410">exciting proposal</a> is
expected to land in Go 1.22 - enhancing the pattern-matching capabilities of
the default HTTP serving multiplexer in the <tt class="docutils literal">net/http</tt> package.</p>
<p>The existing multiplexer (<a class="reference external" href="https://pkg.go.dev/net/http#ServeMux">http.ServeMux</a>) offers rudimentary path matching, but
not much beyond that. This led to a cottage industry of 3rd party libraries …</p><p>An <a class="reference external" href="https://github.com/golang/go/issues/61410">exciting proposal</a> is
expected to land in Go 1.22 - enhancing the pattern-matching capabilities of
the default HTTP serving multiplexer in the <tt class="docutils literal">net/http</tt> package.</p>
<p>The existing multiplexer (<a class="reference external" href="https://pkg.go.dev/net/http#ServeMux">http.ServeMux</a>) offers rudimentary path matching, but
not much beyond that. This led to a cottage industry of 3rd party libraries
to implement more powerful capabilities. I've explored these options in my
<em>REST Servers in Go</em> series, in parts <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-1-standard-library/">1</a>
and <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-2-using-a-router-package/">2</a>.</p>
<p>The new multiplexer in 1.22 is going to significantly bridge the gap from 3rd
party packages by providing advanced matching. In this short post
I'll provide a quick introduction to the new multiplexer (mux).
I'll also revisit the example from the <em>REST Servers in
Go</em> series and compare how the new stdlib mux fares against <tt class="docutils literal">gorilla/mux</tt>.</p>
<img alt="A cartoony go gopher holding a multiplexer" class="align-center" src="https://eli.thegreenplace.net/images/2023/cartoony-gopher-multiplexer.png" />
<div class="section" id="using-the-new-mux">
<h2>Using the new mux</h2>
<p>If you've ever used a 3rd party mux / router package for Go (like
<tt class="docutils literal">gorilla/mux</tt>), using the new standard mux is going to be straightforward and
familiar. Start by reading <a class="reference external" href="https://pkg.go.dev/net/http@master#ServeMux">its documentation</a> - it's short and sweet.</p>
<p>Let's look at a couple of basic usage examples. Our first example demonstrates
some of the new pattern matching capabilities of the mux:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"fmt"</span><span class="w"></span>
<span class="w"> </span><span class="s">"net/http"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"GET /path/"</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprint</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"got path\n"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/task/{id}/"</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">PathValue</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintf</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"handling task with id=%v\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="s">"localhost:8090"</span><span class="p">,</span><span class="w"> </span><span class="nx">mux</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Experienced Go programmers will notice two new features right away:</p>
<ol class="arabic simple">
<li>In the first handler, the HTTP method (<tt class="docutils literal">GET</tt> in this case) is specified
explicitly as part of the pattern. This means that this handler will only
trigger for <tt class="docutils literal">GET</tt> requests to paths beginning with <tt class="docutils literal">/path/</tt>, not for
other HTTP methods.</li>
<li>In the second handler, there's a wildcard in the second path component
- <tt class="docutils literal">{id}</tt>, something that wasn't supported before. The wildcard will match
a single path component and the handler can then access the matched value
through the <tt class="docutils literal">PathValue</tt> method of the request.</li>
</ol>
<p>Since Go 1.22 hasn't been released yet, I recommend running this sample with
<tt class="docutils literal">gotip</tt>. Please see the <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/http-newmux-samples">complete code sample</a>
with full instructions for running this. Let's take this server for a ride:</p>
<div class="highlight"><pre><span></span>$ gotip run sample.go
</pre></div>
<p>And in a separate terminal we can issue some <tt class="docutils literal">curl</tt> calls to test it:</p>
<div class="highlight"><pre><span></span>$ curl localhost:8090/what/
<span class="m">404</span> page not found
$ curl localhost:8090/path/
got path
$ curl -X POST localhost:8090/path/
Method Not Allowed
$ curl localhost:8090/task/f0cd2e/
handling task with <span class="nv">id</span><span class="o">=</span>f0cd2e
</pre></div>
<p>Note how the server rejects a <tt class="docutils literal">POST</tt> request to <tt class="docutils literal">/path/</tt>, while the (default
for <tt class="docutils literal">curl</tt>) <tt class="docutils literal">GET</tt> request is allowed. Note also how the <tt class="docutils literal">id</tt> wildcard gets
assigned a value when the request matches. Once again, I encourage you to review
the <a class="reference external" href="https://pkg.go.dev/net/http@master#ServeMux">documentation of the new ServeMux</a>. You'll learn about additional
capabilities like matching trailing paths to a wildcard with <tt class="docutils literal"><span class="pre">{id}...</span></tt>,
strict matching of a path end with <tt class="docutils literal">{$}</tt>, and other rules.</p>
<p>Particular care in the proposal was given to potential conflicts between
different patterns. Consider this setup:</p>
<div class="highlight"><pre><span></span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/task/{id}/status/"</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">id</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">PathValue</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintf</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"handling task status with id=%v\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">id</span><span class="p">)</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/task/0/{action}/"</span><span class="p">,</span><span class="w"> </span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">PathValue</span><span class="p">(</span><span class="s">"action"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintf</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"handling task 0 with action=%v\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">action</span><span class="p">)</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
</pre></div>
<p>And suppose the server receives a request for <tt class="docutils literal">/task/0/status/</tt> -- which
handler should it go to? It matches both! Therefore, the new <tt class="docutils literal">ServeMux</tt>
documentation meticulously describes the <em>precedence rules</em> for patterns, along
with potential conflicts. In case of a conflict, the registration panics.
Indeed, for the example above we get something like:</p>
<div class="highlight"><pre><span></span>panic: pattern "/task/0/{action}/" (registered at sample-conflict.go:14) conflicts with pattern "/task/{id}/status/" (registered at sample-conflict.go:10):
/task/0/{action}/ and /task/{id}/status/ both match some paths, like "/task/0/status/".
But neither is more specific than the other.
/task/0/{action}/ matches "/task/0/action/", but /task/{id}/status/ doesn't.
/task/{id}/status/ matches "/task/id/status/", but /task/0/{action}/ doesn't.
</pre></div>
<p>The message is detailed and helpful. If we encounter conflicts in complex
registration schemes (especially when patterns are registered in multiple places
in the source code), such details will be much appreciated.</p>
</div>
<div class="section" id="redoing-my-task-server-with-the-new-mux">
<h2>Redoing my task server with the new mux</h2>
<p>The <em>REST Servers in Go</em> series implements a simple server for a task/todo-list
application in Go, using several different approaches. <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-1-standard-library/">Part 1</a>
starts with a "vanilla" standard library approach, and <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-2-using-a-router-package/">Part 2</a>
reimplements the same server using the <a class="reference external" href="https://github.com/gorilla/mux">gorilla/mux</a> router.</p>
<p>Now is a great time to reimplement it once again, but with the enhanced mux
from Go 1.22; it will be particularly interesting to compare the solution to
the one using <tt class="docutils literal">gorilla/mux</tt>.</p>
<p>The full code for this project is <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2021/go-rest-servers/stdlib-newmux">available here</a>.
Let's look at a few representative code samples, starting with the pattern
registration <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p>
<div class="highlight"><pre><span></span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="nx">server</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">NewTaskServer</span><span class="p">()</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"POST /task/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">createTaskHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"GET /task/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">getAllTasksHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"DELETE /task/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">deleteAllTasksHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"GET /task/{id}/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">getTaskHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"DELETE /task/{id}/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">deleteTaskHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"GET /tag/{tag}/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">tagHandler</span><span class="p">)</span><span class="w"></span>
<span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"GET /due/{year}/{month}/{day}/"</span><span class="p">,</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">dueHandler</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>Just like in the <tt class="docutils literal">gorilla/mux</tt> sample, here we use specific HTTP methods
to route requests (with the same path) to different handlers; with the older
<tt class="docutils literal">http.ServeMux</tt>, such matchers had to go to the same handler, which would then
decide what to do based on the method.</p>
<p>Let's also look at one of the handlers:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">ts</span><span class="w"> </span><span class="o">*</span><span class="nx">taskServer</span><span class="p">)</span><span class="w"> </span><span class="nx">getTaskHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"handling get task at %s\n"</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">URL</span><span class="p">.</span><span class="nx">Path</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">id</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">strconv</span><span class="p">.</span><span class="nx">Atoi</span><span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">PathValue</span><span class="p">(</span><span class="s">"id"</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"invalid id"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusBadRequest</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">task</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ts</span><span class="p">.</span><span class="nx">store</span><span class="p">.</span><span class="nx">GetTask</span><span class="p">(</span><span class="nx">id</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusNotFound</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">renderJSON</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">task</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>It extracts the ID value from <tt class="docutils literal"><span class="pre">req.PathValue("id")</span></tt>, similarly to the Gorilla
approach; however, since we don't have a regexp specifying that <tt class="docutils literal">{id}</tt> only
matches integers, we have to pay attention to errors returned from
<tt class="docutils literal">strconv.Atoi</tt>.</p>
<p>All and all, the end result is remarkably similar to the solution that uses
<tt class="docutils literal">gorilla/mux</tt> from <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-2-using-a-router-package/">part 2</a>.
The handlers are much better separated than in the vanilla stdlib approach,
because the mux now can do much more sophisticated routing, without leaving many
of the routing decisions to the handlers themselves.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>"Which router package should I use?" has always been a FAQ for beginner Go
programmers. I believe the common answers to this question will shift after
Go 1.22 is released, as many will find the new stdlib mux sufficient for their
needs without resorting to 3rd party packages.</p>
<p>Others will stick to familiar 3rd party packages, and that's totally fine.
Routers like <tt class="docutils literal">gorilla/mux</tt> still provide more capabilities than the standard
library; on top of it, many Go programmers opt for lightweight frameworks like
Gin, which provide a router but also additional tools for building web backends.</p>
<p>All in all, this is certainly a positive change for all Go users. Making the
standard library more capable is a net positive for the entire community,
whether people use 3rd party packages or stick to just the standard library.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>You may have noticed that these patterns aren't very strict w.r.t
path parts that come after the part we care about (e.g.
<tt class="docutils literal">/task/22/foobar</tt>). This is in line with the rest of the series, but
the new <tt class="docutils literal">http.ServeMux</tt> makes it very easy to restrict the paths with
a trailing <tt class="docutils literal">{$}</tt> wildcard, if needed.</td></tr>
</tbody>
</table>
</div>
static-server: an HTTP server in Go for static content2023-09-16T06:20:00-07:002024-02-25T14:36:58-08:00Eli Benderskytag:eli.thegreenplace.net,2023-09-16:/2023/static-server-an-http-server-in-go-for-static-content/<p>I put together a simple static file server in Go - useful for local testing
of web applications. Check it out at <a class="reference external" href="https://github.com/eliben/static-server">https://github.com/eliben/static-server</a></p>
<p>If you have Go installed on your machine, you don't have to download anything
else; you can run:</p>
<div class="highlight"><pre><span></span>$ go run github.com/eliben/static-server …</pre></div><p>I put together a simple static file server in Go - useful for local testing
of web applications. Check it out at <a class="reference external" href="https://github.com/eliben/static-server">https://github.com/eliben/static-server</a></p>
<p>If you have Go installed on your machine, you don't have to download anything
else; you can run:</p>
<div class="highlight"><pre><span></span>$ go run github.com/eliben/static-server@latest
</pre></div>
<p>And it will start serving the current directory! Run it with <tt class="docutils literal"><span class="pre">-help</span></tt> for
usage information. No configuration files needed - the default is useful
and you can adjust it to your needs using command-line flags.</p>
<p>Obviously, you can also install it once with:</p>
<div class="highlight"><pre><span></span>$ go install github.com/eliben/static-server@latest
</pre></div>
<p>And then just invoke <tt class="docutils literal"><span class="pre">static-server</span></tt> if your <tt class="docutils literal">PATH</tt> is properly set up.</p>
<div class="section" id="why">
<h2>Why</h2>
<p>When developing web applications locally, for basic test cases we can
open an HTML file directly in the browser (using the <a class="reference external" href="https://en.wikipedia.org/wiki/File_URI_scheme">file:/// scheme</a>). However, this is sometimes
insufficient, and in several scenarios it's necessary to properly <em>serve</em> the
HTML (along with its JS and CSS). Some cases where I encountered this are web
applications that use at least one of:</p>
<ul class="simple">
<li>Web workers</li>
<li>Web sockets</li>
<li>WASM</li>
<li>Separate API servers, requiring <a class="reference external" href="https://eli.thegreenplace.net/2023/introduction-to-cors-for-go-programmers/">CORS</a></li>
<li>Loading ES modules from separate files</li>
</ul>
<p>In the past, when I was more active in the Python ecosystem, I used
<tt class="docutils literal">python <span class="pre">-m</span> SimpleHTTPServer <port></tt> quite a bit. While it's nice, it has some
issues too: it's not very configurable, and it requires Python to be installed.</p>
<p>Another option I've used is <a class="reference external" href="https://www.npmjs.com/package/http-server">http-server</a> from the Node.js ecosystem; in
fact, it has served as the inspiration for <tt class="docutils literal"><span class="pre">static-server</span></tt>. You can run it
with <tt class="docutils literal">npx</tt> without installing, and it's also configurable through command-line
flags, without requiring configuration files.</p>
<p>But we can't expect all Go developers to have <tt class="docutils literal">npm</tt> or <tt class="docutils literal">npx</tt> installed.
Moreover, sometimes you want to tweak the server a bit and digging in JavaScript
is not any Go programmer's idea of a good time. Like many tools in that
ecosystem, this Node.js-based HTTP server is all in on dependencies - with 13
of them, it's not easy to understand or modify its code; much of it is split
across multiple helper packages, and making changes can be tricky.</p>
</div>
<div class="section" id="how">
<h2>How</h2>
<p>Spinning up a static file server in Go is very easy - I wrote a
<a class="reference external" href="https://eli.thegreenplace.net/2022/serving-static-files-and-web-apps-in-go/">whole blog post</a> about the
possibilities at some point. The simplest static server to serve the current
working directory is just:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="s">"net/http"</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">port</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">":8080"</span><span class="w"></span>
<span class="w"> </span><span class="nx">handler</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">FileServer</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Dir</span><span class="p">(</span><span class="s">"."</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span><span class="w"> </span><span class="nx">handler</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Having found myself plopping a small <tt class="docutils literal">server.go</tt> with these contents in too
many web projects, I decided enough was enough.
Thus <a class="reference external" href="https://github.com/eliben/static-server">static-server</a> was born.</p>
<p><tt class="docutils literal"><span class="pre">static-server</span></tt> is simple, yet versatile. It will do the right thing
by default, with no flags whatsoever. But you can also use flags to configure
a few aspects, e.g.: the port it serves on, CORS support, serving via TLS,
control how logging is done.</p>
<p><tt class="docutils literal"><span class="pre">static-server</span></tt> is hackable and easy to understand. All the code is in a
single file (with fewer than 200 lines of code, including comments and handling
flags) and there are <em>no dependencies</em> (except one package that is only used
for testing).</p>
<p>I find <tt class="docutils literal"><span class="pre">static-server</span></tt> very useful, and I hope others will too. If you run
into any problems or have questions, open a GitHub issue or send me an email.</p>
</div>
Introduction to CORS for Go programmers2023-09-09T05:50:00-07:002024-03-04T13:22:43-08:00Eli Benderskytag:eli.thegreenplace.net,2023-09-09:/2023/introduction-to-cors-for-go-programmers/<p>From its inception, the Web has been a game of whackamole between people finding
security holes and exploits, and other people plugging these holes and adding
defensive security mechanisms.</p>
<p>One of the busiest arenas in this struggle is the interaction between code
running on one site (via JavaScript embedded in …</p><p>From its inception, the Web has been a game of whackamole between people finding
security holes and exploits, and other people plugging these holes and adding
defensive security mechanisms.</p>
<p>One of the busiest arenas in this struggle is the interaction between code
running on one site (via JavaScript embedded in its page) and other sites;
you may have heard about acronyms like XSS, CSRF, SSRF, SOP and CORS - they
are all related to this dynamic and fascinating aspect of modern computer
security. This post talks specifically about CORS, and what you should know
if you're writing servers in Go.</p>
<div class="section" id="same-origin-policy">
<h2>Same-origin policy</h2>
<p>Our story starts with the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy">Same-origin policy (SOP)</a> -
a mechanism built into browsers that prevents arbitrary access from the
site you're currently browsing to other sites. Suppose you're browsing
<a class="reference external" href="https://catvideos.meow">https://catvideos.meow</a>; while you're doing so, your browser will execute JS
code from that site's pages.</p>
<p>JS can - among other things - fetch
resources from other domains; this is commonly used for images, stats, ads, for
loading other JS modules from CDNs and so on.</p>
<p>But it's also an
inherently unsafe operation, because what if someone injects malicious code
into catvideos.meow that sends requests to <a class="reference external" href="https://yourbank.com">https://yourbank.com</a>! Since the JS
of catvideos.meow is executed by your browser, this is akin to you opening a
new browser window and visiting <a class="reference external" href="https://yourbank.com">https://yourbank.com</a>, including providing any
log-in information and cookies that may already be saved in your browser's
session. That doesn't sound very safe!</p>
<p>This is what the SOP was designed to prevent; generally speaking, except for a
limited set of "safe" (but mostly there for historical reasons) use cases like
fetching images, embedding and submitting a limited set of forms, JS is not
allowed to make cross-origin requests.</p>
<p>A request is considered cross-origin if it's made from origin A to origin B,
and any of the following differ between the origins: protocol, domain and
port (a default port is assumed per protocol, if not explicitly provided):</p>
<img alt="The parts of a domain for CORS" class="align-center" src="https://eli.thegreenplace.net/images/2023/cors-domain-parts.png" />
<p>If the protocol, domain and port match, the request is valid - the path doesn't
matter. Naturally, this is used all the time by JS loading other resources from
its own domain.</p>
</div>
<div class="section" id="local-experiment-to-observe-the-sop-in-action">
<h2>Local experiment to observe the SOP in action</h2>
<p>Let's try a simple experiment to see how this browser protection works;
this only requires a couple of small HTML files with a bit of
JS. Place two HTML files in the same directory; one should be named
<tt class="docutils literal">page.html</tt> and its contents don't matter. The other should be named
<tt class="docutils literal"><span class="pre">do-fetch.html</span></tt>, with these contents:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>Fetch another page<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://127.0.0.1:8080/page.html'</span><span class="w"></span>
<span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">response</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">status</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">error</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"ERROR:"</span><span class="p">,</span><span class="w"> </span><span class="nx">error</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>It attempts to load <tt class="docutils literal">page.html</tt> from a URL (which points to a local machine's
port) via the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch">fetch() API</a>.</p>
<p><strong>First experiment</strong>: run a local static file server in the directory containing
these two HTML files. Feel free to use my <a class="reference external" href="https://github.com/eliben/static-server">static-server</a> project, but any server will do
<a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p>
<div class="highlight"><pre><span></span>$ go install github.com/eliben/static-server@latest
$ ls
do-fetch.html page.html
$ static-server -port 8080 .
2023/09/03 06:02:10.111818 Serving directory "." on http://127.0.0.1:8080
</pre></div>
<p>This serves our two HTML files on local port 8080. Now we can point our
browser to <a class="reference external" href="http://127.0.0.1:8080/do-fetch.html">http://127.0.0.1:8080/do-fetch.html</a> and open the browser console.
There shouldn't be errors, and we should see the printout <tt class="docutils literal">200</tt>, which is
the successful HTTP response from attempting to load <tt class="docutils literal">page.html</tt>.
It succeeds because this is a same-origin <tt class="docutils literal">fetch</tt>, from
<tt class="docutils literal"><span class="pre">http://127.0.0.1:8080</span></tt> to itself.</p>
<p><strong>Second experiment</strong>: while the static server on port 8080 is still running,
run <em>another</em> instance of the server, serving the same directory on a different
port - you'll want to do this in a separate terminal:</p>
<div class="highlight"><pre><span></span>$ ls
do-fetch.html page.html
$ static-server -port 9999 .
2023/09/03 06:12:19.742790 Serving directory "." on http://127.0.0.1:9999
</pre></div>
<p>Now, let's point the browser to <a class="reference external" href="http://127.0.0.1:9999/do-fetch.html">http://127.0.0.1:9999/do-fetch.html</a> and open
the browser console again. The page won't load, and instead you'll see an
error similar to:</p>
<div class="highlight"><pre><span></span>Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
remote resource at http://127.0.0.1:8080/page.html. (Reason: CORS header
‘Access-Control-Allow-Origin’ missing).
</pre></div>
<p>This is the SOP in action. Here's what's going on:</p>
<ul class="simple">
<li>As far as the browser is concerned, a web page at origin
<tt class="docutils literal"><span class="pre">http://127.0.0.1:9999</span></tt> is making a <tt class="docutils literal">fetch</tt> call to origin
<tt class="docutils literal"><span class="pre">http://127.0.0.1:8080</span></tt> (note that this destination is hard-coded in
the source of <tt class="docutils literal"><span class="pre">do-fetch.html</span></tt>).</li>
<li>Since the ports are different, these are considered to be
<em>different origins</em>, and the <tt class="docutils literal">fetch</tt> is a cross-origin request.</li>
<li>By the default SOP, cross-origin requests are blocked.</li>
</ul>
<p>Note that the browser also mentions a CORS header, which is a great segue to
our next topic.</p>
</div>
<div class="section" id="cors">
<h2>CORS</h2>
<p>So what is CORS, and how can it help us make requests to different origins?
The CORS acronym stands for Cross-Origin Resource Sharing, and this is a
good definition <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS">from MDN</a>:</p>
<blockquote>
Cross-Origin Resource Sharing (CORS) is an HTTP-header based mechanism that
allows a server to indicate any origins (domain, scheme, or port) other than
its own from which a browser should permit loading resources.</blockquote>
<p>CORS is a simple protocol between an HTTP server and a browser. When a page
attempts to make a cross-origin request, the browser attaches a special header
to the request with the name <tt class="docutils literal">Origin</tt>; in this header, the browser specifies
the origin from which the request originates.</p>
<p>We can actually observe this if we look at the debug console of the browser in
more detail in our SOP experiment. In the <em>Network</em> tab, we can examine the
exact HTTP request made by the browser to fetch the page from
<tt class="docutils literal"><span class="pre">http://127.0.0.1:8080/page.html</span></tt> when <tt class="docutils literal"><span class="pre">do-fetch.html</span></tt> asked for it.
We should see something like:</p>
<div class="highlight"><pre><span></span>GET /page.html HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: http://127.0.0.1:9999/
Origin: http://127.0.0.1:9999
</pre></div>
<p>The important line here is the last one: it tells the server which origin
the request is coming from.</p>
<p>We can also examine the server's response, in which we'll see that the server
does not include a special header named <tt class="docutils literal"><span class="pre">Access-Control-Allow-Origin</span></tt>. Since
this header is not in the response, the browser assumes that the server doesn't
support CORS from the specified origin, and this results in the error we've seen
above.</p>
<p>To complete a successful cross-origin request, the server has to approve the
request explicitly by returning an <tt class="docutils literal"><span class="pre">Access-Control-Allow-Origin</span></tt> header. The
value of the header should be either the origin named in the request's
<tt class="docutils literal">Origin</tt> header, or the special value <tt class="docutils literal">*</tt> which means "all origins
accepted".</p>
<img alt="Diagram illustrating the exchange of the CORS protocol" class="align-center" src="https://eli.thegreenplace.net/images/2023/cors-protocol.png" />
<p>To see this in action, it's time for another experiment; let's write a simple
Go server that supports cross-origin requests.</p>
</div>
<div class="section" id="a-sample-go-server-with-cors-support">
<h2>A sample Go server with CORS support</h2>
<p>Leaving static file serving behind, let's move closer towards what CORS is
actually used for: protecting access to APIs from unknown origins. Here's a
simple Go server that serves a very basic API endpoint at <tt class="docutils literal">/api</tt>, returning a
hard-coded JSON value:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">apiHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span><span class="w"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintln</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">`{"message": "hello"}`</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">port</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">":8080"</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api"</span><span class="p">,</span><span class="w"> </span><span class="nx">apiHandler</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span><span class="w"> </span><span class="nx">mux</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This server should be started locally; Here's a somewhat modified HTML file with
JS making a CORS request to this endpoint, assuming the server runs on local
port 8080:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>Access API through CORS<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://localhost:8080/api'</span><span class="w"></span>
<span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">response</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="ne">Error</span><span class="p">(</span><span class="s1">'Failed to fetch data'</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">data</span><span class="p">.</span><span class="nx">message</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">error</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="s2">"ERROR: "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">error</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>Assuming this code is saved locally in <tt class="docutils literal"><span class="pre">access-through-cors.html</span></tt>, we will
serve it with <tt class="docutils literal"><span class="pre">static-server</span></tt> on port 9999, as before:</p>
<div class="highlight"><pre><span></span>$ static-server -port 9999 .
2023/09/03 08:01:22.413757 Serving directory "." on http://127.0.0.1:9999
</pre></div>
<p>When we open <a class="reference external" href="http://127.0.0.1:9999/access-through-cors.html">http://127.0.0.1:9999/access-through-cors.html</a> in the browser,
we'll see the CORS error again:</p>
<div class="highlight"><pre><span></span>Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
remote resource at http://127.0.0.1:8080/api. (Reason: CORS header
‘Access-Control-Allow-Origin’ missing).
</pre></div>
<p>Indeed, our server doesn't support CORS yet! This is an important point to
emphasize - a server oblivious to CORS means it doesn't support it.
In other words, CORS is "opt-in". Since our server doesn't check for the
<tt class="docutils literal">Origin</tt> header and doesn't return the expected CORS headers back to the
client, the browser assumes that the cross-origin request is denied, and returns
an error to the HTML page <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>.</p>
<p>Let's fix that, and implement CORS in our server. It's customary to do it
as middleware that wraps the HTTP handler. Here's a simple approach:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">originAllowlist</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://127.0.0.1:9999"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://cats.com"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://safe.frontend.net"</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">checkCORS</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">origin</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">slices</span><span class="p">.</span><span class="nx">Contains</span><span class="p">(</span><span class="nx">originAllowlist</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Access-Control-Allow-Origin"</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Add</span><span class="p">(</span><span class="s">"Vary"</span><span class="p">,</span><span class="w"> </span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">next</span><span class="p">.</span><span class="nx">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">checkCORS</tt> is <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-5-middleware/">standard Go middleware</a>.
It wraps any HTTP handler and adds CORS logic on top; here's how it works:</p>
<ul class="simple">
<li>It checks if the <tt class="docutils literal">Origin</tt> header is present in the request (the header's
<tt class="docutils literal">Get</tt> method will return an empty string for a missing header).</li>
<li>If yes, it checks its value; if it's in an allow-list of authorized origins,
the origin is parroted back to the client in the
<tt class="docutils literal"><span class="pre">Access-Control-Allow-Origin</span></tt> header of the response.</li>
<li>We also set <tt class="docutils literal">Vary: Origin</tt> in the response to avoid problems with caching
proxies between the server and the client (see
<a class="reference external" href="https://fetch.spec.whatwg.org/#cors-protocol-and-http-caches">this section in the fetch standard</a> for more details)</li>
<li>If there is no <tt class="docutils literal">Origin</tt> header, or the origin value is not in our
allow-list, the middleware doesn't change the response headers in any way.
As we saw before, this is equivalent to saying "I don't support cross-origin
requests from that origin".</li>
</ul>
<p>Obviously, the allow-list solution presented here is ad-hoc, and you are free
to implement your own. Some API endpoints want to be truly public and support
cross-origin requests from <em>any</em> domain. In such cases, one can just hard-code
<tt class="docutils literal"><span class="pre">Access-Control-Allow-Origin:</span> *</tt> in all responses, without additional logic.
In this case the <tt class="docutils literal">Vary</tt> header isn't required either.</p>
<p>Now that we have the middleware in place, we have to hook it into our server;
let's wrap the top-level router, so <tt class="docutils literal">checkCORS</tt> applies to all endpoints we
may add to the server in the future:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">port</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">":8080"</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api"</span><span class="p">,</span><span class="w"> </span><span class="nx">apiHandler</span><span class="p">)</span><span class="w"></span>
<span class="hll"><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span><span class="w"> </span><span class="nx">checkCORS</span><span class="p">(</span><span class="nx">mux</span><span class="p">))</span><span class="w"></span>
</span><span class="p">}</span><span class="w"></span>
</pre></div>
<p>If we kill the old server occupying port 8080 and run this one instead,
re-loading <tt class="docutils literal"><span class="pre">access-through-cors.html</span></tt> we'll see different results: the
page shows "hello" and there are no errors in the console. The CORS request
succeeded! Let's examine the response headers:</p>
<div class="highlight"><pre><span></span>HTTP/1.1 200 OK
<span class="hll">Access-Control-Allow-Origin: http://127.0.0.1:9999
</span>Content-Type: application/json
<span class="hll">Vary: Origin
</span>Date: Sun, 03 Sep 2023 16:33:00 GMT
Content-Length: 21
</pre></div>
<p>The custom headers set by our middleware are highlighted; the request
was made by a page served on local port 9999, and this is in the <tt class="docutils literal">Origin</tt>
header sent by the browser. Therefore, our response headers permit the browser
to communicate the data back to the client code and finish without errors.
As an exercise, modify the <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2023/cors/go-server-cors/serve-api-with-cors.go">code of our CORS middleware</a>
to set <tt class="docutils literal">*</tt> instead of a specific origin, then re-run the server and client
page, and examine the response header again.</p>
</div>
<div class="section" id="preflight-requests">
<h2>Preflight requests</h2>
<p>As we've seen, when a page issues a cross-origin request, the browser obliges,
but withholds any response details from the fetching code unless the server
explicitly agreed to receive the request via CORS. This can be worrisome,
though; what if the request itself causes something unsafe to happen on the
server?</p>
<p>This is what <em>preflight</em> requests are for; for some HTTP requests that aren't
deemed inherently safe, a browser will first send a special <tt class="docutils literal">OPTIONS</tt> request
(called "preflight") to double check that the server is ready for this kind
of request from the specific origin. Only if answered in the affirmative, the
browser will then send the actual HTTP request.</p>
<p>The terminology here gets a bit confusing.
<a class="reference external" href="https://www.w3.org/TR/2020/SPSD-cors-20200602/#simple-method">The old CORS standard</a> defines
<em>simple requests</em> as those that don't require preflight, but the <a class="reference external" href="https://fetch.spec.whatwg.org/#cors-protocol-and-http-caches">new fetch
standard that defines CORS</a> doesn't use
this term. Generally, <tt class="docutils literal">GET</tt>, <tt class="docutils literal">HEAD</tt> and <tt class="docutils literal">POST</tt> requests restricted to
certain headers and content types are considered <em>simple</em>; for the full
definition, see the linked standards. Anything that isn't simple requires a
preflight <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a>.</p>
<p>The protocol goes as follows:</p>
<ul class="simple">
<li>If the client tries to send a request that's not "simple", the browser will
first send an <tt class="docutils literal">OPTIONS</tt> request with the <tt class="docutils literal"><span class="pre">Access-Control-Request-Method</span></tt>
header. The value of this header is the actual request method the client
wants. The <tt class="docutils literal">OPTIONS</tt> request also carries an <tt class="docutils literal">Origin</tt> header specifying
which origin this is coming from.</li>
<li>If the server supports the specified method from this origin, it returns
a successful response with the <tt class="docutils literal"><span class="pre">Access-Control-Allow-Methods</span></tt> header where
it lists the supported methods from this origin.</li>
<li>The browser processes the response to <tt class="docutils literal">OPTIONS</tt> and determines whether the
request it asked to send on behalf of the origin is on the allow-list;
if yes, it then sends the actual request the client made. Note that since HTTP
is stateless, the actual request will also follow the CORS protocol.</li>
</ul>
<img alt="Diagram illustrating the exchange of the CORS protocol with preflight" class="align-center" src="https://eli.thegreenplace.net/images/2023/cors-preflight-protocol.png" />
<p>There's another feature of preflight requests which I'm not going to cover in
detail here,
but it's easy enough to implement if needed: permissions for special headers.
Preflight requests not only protect servers from potentially unsafe methods,
but also from <a class="reference external" href="https://fetch.spec.whatwg.org/#cors-safelisted-request-header">potentially unsafe headers</a>.
If the client tries to send a cross-origin request with such headers, the
browser will send a preflight with the <tt class="docutils literal"><span class="pre">Access-Control-Request-Headers</span></tt> header
listing these headers; the server has to reply with
<tt class="docutils literal"><span class="pre">Access-Control-Allow-Headers</span></tt> in order for the protocol to succeed.</p>
</div>
<div class="section" id="adding-preflight-support-to-our-go-server">
<h2>Adding preflight support to our Go server</h2>
<p>Before working on the server's code, let's see how the browser sends preflight
requests on behalf of a <tt class="docutils literal">fetch</tt> call. We'll update the JS code in
our HTML page just a bit:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://localhost:8080/api'</span><span class="w"></span>
<span class="hll"><span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">method</span><span class="o">:</span><span class="w"> </span><span class="s1">'DELETE'</span><span class="p">})</span><span class="w"></span>
</span><span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">response</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="ne">Error</span><span class="p">(</span><span class="s1">'Failed to fetch data'</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">data</span><span class="p">.</span><span class="nx">message</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">error</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="s2">"ERROR: "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">error</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
</pre></div>
<p>With the old CORS server (that doesn't support preflight requests yet) still
running on port 8080, when we open this page in the browser served at
127.0.0.1:9999, we'll see an error:</p>
<div class="highlight"><pre><span></span>Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
remote resource at http://localhost:8080/api. (Reason: Did not find method in
CORS header ‘Access-Control-Allow-Methods’).
</pre></div>
<p>Diving deeper, we find that the browser sent an <tt class="docutils literal">OPTIONS</tt> request to the
server with the following relevant headers:</p>
<div class="highlight"><pre><span></span>Access-Control-Request-Method: DELETE
Origin: http://127.0.0.1:9999
</pre></div>
<p>This means "hey server, some code at origin <tt class="docutils literal">127.0.0.1:9999</tt> wants to send
you a <tt class="docutils literal">DELETE</tt> request, are you cool with that?"</p>
<p>Did our server reply? Yes, with the same response it sent for the <tt class="docutils literal">GET</tt>
request in the previous example:</p>
<div class="highlight"><pre><span></span>HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://127.0.0.1:9999
Content-Type: application/json
Vary: Origin
Date: Sun, 03 Sep 2023 21:34:03 GMT
Content-Length: 21
</pre></div>
<p>That's because we haven't actually restricted the method in our Go server: it
answers the same response to <em>all</em> methods - in this case <tt class="docutils literal">OPTIONS</tt>! Since
the browser sent our server a preflight for <tt class="docutils literal">DELETE</tt>, it expected the server
to reply with <tt class="docutils literal"><span class="pre">Access-Control-Allow-Methods</span></tt> that lists <tt class="docutils literal">DELETE</tt>. The
server didn't, so the browser aborted the procedure and returned an error to the
client (<em>without</em> actually sending the <tt class="docutils literal">DELETE</tt> request itself).</p>
<p>Let's now fix that, by implementing preflight in our server. We'll start with
a helper function that reports whether the given request is a preflight request:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">isPreflight</span><span class="p">(</span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Method</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"OPTIONS"</span><span class="w"> </span><span class="o">&&</span><span class="w"></span>
<span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Origin"</span><span class="p">)</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">""</span><span class="w"> </span><span class="o">&&</span><span class="w"></span>
<span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Access-Control-Request-Method"</span><span class="p">)</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="s">""</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>It's important to note that all three conditions have to be true for the
request to be considered preflight. Next, we'll modify our <tt class="docutils literal">checkCORS</tt>
middleware to support preflights:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">originAllowlist</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://127.0.0.1:9999"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://cats.com"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"http://safe.frontend.net"</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">var</span><span class="w"> </span><span class="nx">methodAllowlist</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"GET"</span><span class="p">,</span><span class="w"> </span><span class="s">"POST"</span><span class="p">,</span><span class="w"> </span><span class="s">"DELETE"</span><span class="p">,</span><span class="w"> </span><span class="s">"OPTIONS"</span><span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">checkCORS</span><span class="p">(</span><span class="nx">next</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="p">)</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Handler</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">HandlerFunc</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">isPreflight</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">origin</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">method</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Access-Control-Request-Method"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">slices</span><span class="p">.</span><span class="nx">Contains</span><span class="p">(</span><span class="nx">originAllowlist</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">slices</span><span class="p">.</span><span class="nx">Contains</span><span class="p">(</span><span class="nx">methodAllowlist</span><span class="p">,</span><span class="w"> </span><span class="nx">method</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Access-Control-Allow-Origin"</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Access-Control-Allow-Methods"</span><span class="p">,</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">methodAllowlist</span><span class="p">,</span><span class="w"> </span><span class="s">", "</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Add</span><span class="p">(</span><span class="s">"Vary"</span><span class="p">,</span><span class="w"> </span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Not a preflight: regular request.</span><span class="w"></span>
<span class="w"> </span><span class="nx">origin</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Header</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">slices</span><span class="p">.</span><span class="nx">Contains</span><span class="p">(</span><span class="nx">originAllowlist</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Access-Control-Allow-Origin"</span><span class="p">,</span><span class="w"> </span><span class="nx">origin</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Add</span><span class="p">(</span><span class="s">"Vary"</span><span class="p">,</span><span class="w"> </span><span class="s">"Origin"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">next</span><span class="p">.</span><span class="nx">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>If we run this updated server on port 8080 and invoke the HTML page that
does a <tt class="docutils literal">fetch</tt> with <tt class="docutils literal">method: 'DELETE'</tt> again, the request will be
successful. The server now has a tailored reply for the <tt class="docutils literal">OPTIONS</tt> preflight
request:</p>
<div class="highlight"><pre><span></span>HTTP/1.1 200 OK
<span class="hll">Access-Control-Allow-Methods: GET, POST, DELETE, OPTIONS
</span><span class="hll">Access-Control-Allow-Origin: http://127.0.0.1:9999
</span>Content-Type: application/json
Vary: Origin
Date: Sun, 03 Sep 2023 13:12:29 GMT
Content-Length: 21
</pre></div>
<p>The browser then proceeds to send the <tt class="docutils literal">DELETE</tt> request itself:</p>
<div class="highlight"><pre><span></span>DELETE /api HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: http://127.0.0.1:9999/
Origin: http://127.0.0.1:9999
</pre></div>
<p>Which gets a successful reply:</p>
<div class="highlight"><pre><span></span>HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://127.0.0.1:9999
Content-Type: application/json
Vary: Origin
Date: Sun, 03 Sep 2023 13:12:29 GMT
Content-Length: 21
</pre></div>
</div>
<div class="section" id="cookies-and-cors">
<h2>Cookies and CORS</h2>
<p>At the beginning of the post we discussed how sending cookies on behalf of
the visiting browser is one of the main security issues the SOP and CORS try
to address. Now it's time to discuss this in more detail.</p>
<p>Let's go back to our server and have it set a cookie when a certain path is
accessed. Our <tt class="docutils literal">main</tt> function becomes:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">port</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="s">":8080"</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">NewServeMux</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/api"</span><span class="p">,</span><span class="w"> </span><span class="nx">apiHandler</span><span class="p">)</span><span class="w"></span>
<span class="hll"><span class="w"> </span><span class="nx">mux</span><span class="p">.</span><span class="nx">HandleFunc</span><span class="p">(</span><span class="s">"/getcookie"</span><span class="p">,</span><span class="w"> </span><span class="nx">getCookieHandler</span><span class="p">)</span><span class="w"></span>
</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span><span class="w"> </span><span class="nx">checkCORS</span><span class="p">(</span><span class="nx">mux</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>And <tt class="docutils literal">getCookieHandler</tt> is:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">getCookieHandler</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">r</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Set-Cookie"</span><span class="p">,</span><span class="w"> </span><span class="s">"somekey=somevalue"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Fprintln</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">`{"message": "you're welcome"}`</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Very simple: everyone visiting the <tt class="docutils literal">/getcookie</tt> route gets a cookie! If we
run this server on port 8080 as usual and visit <a class="reference external" href="http://http://127.0.0.1:8080/getcookie">http://http://127.0.0.1:8080/getcookie</a>,
we should see the cookie sent in the response header:</p>
<div class="highlight"><pre><span></span>HTTP/1.1 200 OK
<span class="hll">Set-Cookie: somekey=somevalue
</span>Date: Sun, 03 Sep 2023 13:25:09 GMT
Content-Length: 30
Content-Type: text/plain; charset=utf-8
</pre></div>
<p>Note: this isn't a CORS request; this is the browser accessing the server
directly. Opening the developer console ("Storage" tab), we should be able to
see this cookie is now associated with 127.0.0.1:8080, something like:</p>
<img alt="Showing the cookie in the developer tools storage tab" class="align-center" src="https://eli.thegreenplace.net/images/2023/cors-cookie-storage.png" />
<p>If we refresh the page, we'll notice that the browser now sends a <tt class="docutils literal">Cookie</tt>
header with this cookie in requests to 127.0.0.1:8080 - as expected!</p>
<p>Next, let's try to access <tt class="docutils literal">/api</tt> again from our HTML page served on a
different origin (port 9999):</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>CORS with credentials<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://localhost:8080/api'</span><span class="w"></span>
<span class="hll"><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">credentials</span><span class="o">:</span><span class="w"> </span><span class="s2">"include"</span><span class="p">})</span><span class="w"></span>
</span><span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">response</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">response</span><span class="p">.</span><span class="nx">ok</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">response</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="ne">Error</span><span class="p">(</span><span class="s1">'Failed to fetch data'</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">data</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="nx">data</span><span class="p">.</span><span class="nx">message</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">error</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">innerHTML</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="s2">"ERROR: "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">error</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>This is where things get interesting; our browser has a cookie associated
with <tt class="docutils literal">127.0.0.1:8080</tt>, and now a different origin makes a request to this
domain inside our browser.</p>
<p><tt class="docutils literal">fetch</tt> won't set cookies by default, and it needs to be told to do so
explicitly (this is yet another security mechanism). The highlighted line
shows how to do this, by adding a <tt class="docutils literal">credentials</tt> options set to <tt class="docutils literal">true</tt>.
When we serve this page on <a class="reference external" href="http://127.0.0.1:9999/getcookie.html">http://127.0.0.1:9999/getcookie.html</a>, we'll see
that the cookie is sent in the request with this header:</p>
<div class="highlight"><pre><span></span>Cookie: somekey=somevalue
</pre></div>
<p>But there's a CORS error in the console, and the browser returns an error
to <tt class="docutils literal">fetch</tt>:</p>
<div class="highlight"><pre><span></span>Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
remote resource at http://localhost:8080/api. (Reason: expected ‘true’ in CORS
header ‘Access-Control-Allow-Credentials’).
</pre></div>
<p>This is because our server doesn't support credentials for CORS yet! As the
error suggests, to signal that credentials are supported, the server has to
set a special header named <tt class="docutils literal"><span class="pre">Access-Control-Allow-Credentials</span></tt> to <tt class="docutils literal">true</tt>:</p>
<div class="highlight"><pre><span></span><span class="nx">w</span><span class="p">.</span><span class="nx">Header</span><span class="p">().</span><span class="nx">Set</span><span class="p">(</span><span class="s">"Access-Control-Allow-Credentials"</span><span class="p">,</span><span class="w"> </span><span class="s">"true"</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>If we rerun the server with this header set, the CORS request with the cookie
succeeds.</p>
<p>Once again, note that for "simple" requests the browser <em>does</em> send the request
with the cookie to the server; it just refuses to get any reply back to the
<tt class="docutils literal">fetch</tt> unless the server explicitly accepts credentials over CORS by
returning a special header. It's the server's job to ensure that nothing unsafe
happens as a result of an unauthorized cross-origin request. For non-simple
methods, the browser will expect <tt class="docutils literal"><span class="pre">Access-Control-Allow-Credentials</span></tt> to be
set on the response to a preflight request, and the actual request won't have
the cookie unless this condition is unsatisfied.</p>
</div>
<div class="section" id="next-steps">
<h2>Next steps</h2>
<p>This post is an introduction to CORS for Go programmers. It doesn't cover all
the aspects and details of CORS, but should be a good foundation for finding
out more, if desired. For additional resources:</p>
<ul class="simple">
<li>The <a class="reference external" href="https://fetch.spec.whatwg.org">fetch standard</a> is the authoritative
definition of CORS.</li>
<li><a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS">MDN articles</a>
are always great.</li>
<li>The web.dev articles on <a class="reference external" href="https://web.dev/same-origin-policy/">SOP</a> and
<a class="reference external" href="https://web.dev/cross-origin-resource-sharing/">CORS</a> are also
recommended.</li>
</ul>
<p>Finally, it's unlikely that you'll have to roll your own CORS implementation.
Popular Go web frameworks like Gin and Echo have CORS middleware built-in,
and projects like <a class="reference external" href="https://github.com/rs/cors">rs/cors</a> provide a
framework-agnostic solution.</p>
</div>
<div class="section" id="code">
<h2>Code</h2>
<p>All the Go and HTML code for this post's samples and experiments is available
<a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/cors">on GitHub</a>.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td><p class="first">When doing experiments involving <tt class="docutils literal">fetch</tt> and other non-trivial JS,
it's strongly recommended to actually <em>serve</em> the HTML files locally,
rather than just opening them with <tt class="docutils literal"><span class="pre">file:///</span></tt> in the browser.
Specifically for CORS, <tt class="docutils literal"><span class="pre">file:///</span></tt> has some additional nuances (e.g.
the <tt class="docutils literal">Origin</tt> header is set to <tt class="docutils literal">null</tt>).</p>
<p>Note also that having different ports on 127.0.0.1 is sufficient to
demonstrate the topics of this post, because ports count in the
definition of "origin".</p>
<p>An alternative is to use the system's <tt class="docutils literal">/etc/hosts</tt> configuration file
to define domain aliases for 127.0.0.1, and run our static server with
<tt class="docutils literal">sudo</tt> to enable serving on port 80. This provides a slightly more
realistic emulation, like accessing <a class="reference external" href="http://foo.domain">http://foo.domain</a> from
<a class="reference external" href="http://bar.domain">http://bar.domain</a>, since the browser is oblivious to
domain aliases (it will even consider <tt class="docutils literal">localhost</tt> and <tt class="docutils literal">127.0.0.1</tt>
distinct for the purposes of CORS).</p>
<p class="last">You're free to do so as an exercise, but having different ports to
represents different origins is generally sufficient for our needs.</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td><p class="first">Note that our Go server still returns a valid JSON response on the
<tt class="docutils literal">/api</tt> endpoint, and the browser gets this response back. However,
the browser won't share it with the client <tt class="docutils literal">fetch()</tt> call, reporting
an error instead.</p>
<p>In fact, if we just <tt class="docutils literal">curl</tt> to <tt class="docutils literal"><span class="pre">http://127.0.0.1:8080/api</span></tt> while
the server is running, we'll get the data back. The CORS mechanism is
a browser feature, not part of the actual HTTP protocol.</p>
<p>This highlights a very important point: while CORS is part of a security
solution, it's absolutely unsuitable as the main (or only) security
mechanism. If you expose an API endpoint on the public internet, clients
<em>will</em> be able to access it. Browsers will block cross-origin requests
from client-side JavaScript, but that's about it.</p>
<p>If you're not actually interested in your endpoint
being public, you should use a real <a class="reference external" href="https://eli.thegreenplace.net/2021/rest-servers-in-go-part-6-authentication/">authentication solution</a>.</p>
<p class="last">And if your server will dutifully execute a <tt class="docutils literal">DELETE</tt> request from
any client on the internet and destroy critical records - you're going
to have a bad time. Don't forget that HTTP is stateless, and the client
is not required to send you a preflight request before a <tt class="docutils literal">DELETE</tt>;
as a matter of fact, all these requests can be easily spoofed using
non-browser clients.</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>You may wonder why <tt class="docutils literal">POST</tt> is considered to be safe; unfortunately, it's
not a good technical reason but rather
backward-compatibility. Forms do submits via <tt class="docutils literal">POST</tt> and this is
something that worked historically, so <tt class="docutils literal">CORS</tt> couldn't interfere
with that. In all fairness, it's a best practice to use
<a class="reference external" href="https://en.wikipedia.org/wiki/Cross-site_request_forgery">CSRF protection</a>
in forms anyway, so there's already a security mechanism applied.</td></tr>
</tbody>
</table>
</div>
RPC-based plugins in Go2023-03-28T20:05:00-07:002023-03-29T03:08:49-07:00Eli Benderskytag:eli.thegreenplace.net,2023-03-28:/2023/rpc-based-plugins-in-go/<p>This post is the next installment in my <a class="reference external" href="https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures">plugin series</a>.
<a class="reference external" href="https://eli.thegreenplace.net/2021/plugins-in-go/">The last post</a>
discussed the two main approaches to developing plugins in Go: compile-time
plugins and run-time plugins. For run-time plugins, the post described how to
use <tt class="docutils literal"><span class="pre">-buildmode=plugin</span></tt> and shared libraries to load plugins at runtime, but
also hinted …</p><p>This post is the next installment in my <a class="reference external" href="https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures">plugin series</a>.
<a class="reference external" href="https://eli.thegreenplace.net/2021/plugins-in-go/">The last post</a>
discussed the two main approaches to developing plugins in Go: compile-time
plugins and run-time plugins. For run-time plugins, the post described how to
use <tt class="docutils literal"><span class="pre">-buildmode=plugin</span></tt> and shared libraries to load plugins at runtime, but
also hinted at an alternative approach that uses separate processes and RPC.</p>
<p>Now it's time to discuss the RPC approach in more detail, using the
<a class="reference external" href="https://github.com/hashicorp/go-plugin">hashicorp/go-plugin</a> package. Here's
a sketch of how such plugins work:</p>
<ol class="arabic simple">
<li>Each plugin is a separate Go binary, built using some code shared with the
main application.</li>
<li>The main application loads plugins by running their binaries as sub-processes.</li>
<li>The main application talks to plugins via RPC to access their functionality.</li>
</ol>
<p>We'll start by explaining how the <tt class="docutils literal"><span class="pre">go-plugin</span></tt> package works and how it helps
us write plugins. Then I'll present a re-implementation of the <em>htmlize</em> program
we've been using throughout the plugin series, this time using <tt class="docutils literal"><span class="pre">go-plugin</span></tt>.</p>
<div class="section" id="the-go-plugin-package">
<h2>The go-plugin package</h2>
<p><a class="reference external" href="https://github.com/hashicorp/go-plugin">go-plugin</a> was developed by
HashiCorp - a powerhouse of Go-based tooling, and has been used in production
by many of their tools (like Terraform and Vault) for years. This is one if its
greatest strengths - it's battle-tested.</p>
<p>The basic idea behind <tt class="docutils literal"><span class="pre">go-plugin</span></tt> is a run-time plugin system wherein each
plugin is a separate binary and runs in its own OS process.</p>
<img alt="A plugin in its on OS process talking RPC with other plugins in their separate OS processes" class="align-center" src="https://eli.thegreenplace.net/images/2023/plugin-rpc-binaries.png" />
<p><tt class="docutils literal"><span class="pre">go-plugin</span></tt> lets us pick which RPC mechanism to use; it supports
<tt class="docutils literal">net/rpc</tt> and gRPC out of the box. Therefore, its API is a bit odd at first
sight. Specifically, we are expected to define our own RPC methods both for
the server (plugin) and the client (main application), and register them with
<tt class="docutils literal"><span class="pre">go-plugin</span></tt> by implementing its <a class="reference external" href="https://pkg.go.dev/github.com/hashicorp/go-plugin#Plugin">Plugin interface</a>.</p>
<p>This leaves one thinking - "wait, so what does <tt class="docutils literal"><span class="pre">go-plugin</span></tt> give me, anyway? If
I have to implement my own RPC, do I really need this helper package?" - which
is a valid question. <tt class="docutils literal"><span class="pre">go-plugin</span></tt> provides several important capabilities,
however:</p>
<ul class="simple">
<li>Handles the actual network connection between a client and multiple servers:
supporting Unix domain sockets on Linux <a class="reference external" href="https://eli.thegreenplace.net/2019/unix-domain-sockets-in-go/">for performance</a>, TCP
elsewhere.</li>
<li>When the client launches a plugin server, <tt class="docutils literal"><span class="pre">go-plugin</span></tt> handles <em>discovery</em>
- figuring out which port/file the server is listening on, and establishing
the connection. This also includes verifying that the launched binary is
the right plugin, meant for this program and not something else.</li>
<li>Supports protocol versioning, which ensures that the main binary doesn't try
to talk to plugins that are "too old".</li>
<li>Supports liveness pings to plugins.</li>
<li>Can set up mTLS between the client and servers (useful when the plugin runs
on a different machine).</li>
<li>Handles redirection of plugin stdin/stdout streams and logs back to the main
process.</li>
<li>Allows having multiple logical plugins reside in the same binary/process,
each with its own RPC interface. All plugins share a single connection to the
main process via connection multiplexing that <tt class="docutils literal"><span class="pre">go-plugin</span></tt> implements. This
also allows plugins to call back into the main application - more on this
later.</li>
</ul>
</div>
<div class="section" id="htmlize-plugins-with-go-plugin">
<h2>htmlize plugins with go-plugin</h2>
<p>Let's rebuild our <tt class="docutils literal">htmlize</tt> tool that we've been using for this demonstration
since the <a class="reference external" href="https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures">original plugins post</a>
(including the <a class="reference external" href="https://eli.thegreenplace.net/2021/plugins-in-go/">Go version</a>), this time using
<tt class="docutils literal"><span class="pre">go-plugin</span></tt>. The full code for this version is <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-plugin-htmlize-rpc">on GitHub</a>.
We will examine it using the <a class="reference external" href="https://eli.thegreenplace.net/2012/08/07/fundamental-concepts-of-plugin-infrastructures">fundamental concepts of plugins</a>.</p>
<p><strong>Discovery and registration:</strong> since plugins are just binaries that can be
found anywhere, <tt class="docutils literal"><span class="pre">go-plugin</span></tt> doesn't prescribe what approach to take here. It
only provides a <tt class="docutils literal">Discover</tt> function which is a basic wrapper around a
filesystem glob pattern. In our code, the <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2023/go-plugin-htmlize-rpc/plugin/manager.go">Manager type</a> takes a
path where it will look for plugin binaries, and treats each file in that
directory as a potential plugin.</p>
<p><tt class="docutils literal"><span class="pre">go-plugin</span></tt> does provide tools to ensure that a loaded binary is, in fact, a
plugin for the right application. When creating a new <tt class="docutils literal"><span class="pre">go-plugin</span></tt> client, we
have to pass in a <a class="reference external" href="https://pkg.go.dev/github.com/hashicorp/go-plugin#HandshakeConfig">HandshakeConfig</a>, which
has to match between the application and the plugin. This helps ensure that we
don't attempt to load a plugin meant for another application, or for a different
version of this application.</p>
<p>As described earlier, at this point <tt class="docutils literal"><span class="pre">go-plugin</span></tt> takes over; it launches the
plugin in a subprocess, connects to its stdout to discover which address the
plugin server is listening on (could be a Unix domain socket or a TCP socket,
based on OS), and then sets up the RPC. The main application (client) is now
ready to invoke RPCs in the plugin (server), based on the agreed-upon interface.</p>
<p><strong>Application hooks:</strong> the central communication point between a plugin and an
application with <tt class="docutils literal"><span class="pre">go-plugin</span></tt> is the plugin's <em>exposed interface</em>. In our
case, the interface is:</p>
<div class="highlight"><pre><span></span><span class="c1">// Htmlizer is the interface plugins have to implement. To avoid calling the</span><span class="w"></span>
<span class="c1">// plugin for roles it doesn't support, it has to tell the plugin managers</span><span class="w"></span>
<span class="c1">// which roles it wants to be invoked on by implementing the Hooks() method.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">Htmlizer</span><span class="w"> </span><span class="kd">interface</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Hooks returns a list of the hooks this plugin wants to register.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Hooks can have one of the following forms:</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// * "contents": the plugin's ProcessContents method will be called on</span><span class="w"></span>
<span class="w"> </span><span class="c1">// the post's complete contents.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// * "role:NN": the plugin's ProcessRole method will be called with role=NN</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and the role's value when a :NN: role is encountered in the</span><span class="w"></span>
<span class="w"> </span><span class="c1">// input.</span><span class="w"></span>
<span class="w"> </span><span class="nx">Hooks</span><span class="p">()</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ProcessRole is called on roles the plugin requested in the list returned</span><span class="w"></span>
<span class="w"> </span><span class="c1">// by Hooks(). It takes the role name, role value in the input and the post</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and should return the transformed role value.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ProcessRole</span><span class="p">(</span><span class="nx">role</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"></span>
<span class="w"> </span><span class="c1">// ProcessContents is called on the entire post contents, if requested in</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Hooks(). It takes the contents and the post and should return the</span><span class="w"></span>
<span class="w"> </span><span class="c1">// transformed contents.</span><span class="w"></span>
<span class="w"> </span><span class="nx">ProcessContents</span><span class="p">(</span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This interface is presented to the application via a RPC mechanism, so code
in the application simply invokes these methods on a value implementing the
interface; <tt class="docutils literal"><span class="pre">go-plugin</span></tt> translates this to RPC calls behind the scenes <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
<p>I recommend you to carefully read the comments on the <tt class="docutils literal">Htmlizer</tt> interface;
they describe an interesting nuance w.r.t. application hooks. In our <tt class="docutils literal">htmlize</tt>
application, we want plugins to register for specific text "roles". If a plugin
didn't register for a role, we don't want to invoke it when the role is
encountered - it's wasteful to call N RPCs for N plugins for each role, when in
reality at most one plugin likely cares about any given role.</p>
<p><tt class="docutils literal"><span class="pre">go-plugin</span></tt> does not provide built-in support to handle this conditional
registration. A plugin exposes an interface via RPC, and that's it. But it turns
out to be fairly easy to implement in a custom way, as our <tt class="docutils literal">Hooks</tt> method
demonstrates. Here is how our plugin <tt class="docutils literal">Manager</tt> type handles this; first the
<tt class="docutils literal">Manager</tt> type itself:</p>
<div class="highlight"><pre><span></span><span class="kd">type</span><span class="w"> </span><span class="nx">Manager</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">roleHooks</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">Htmlizer</span><span class="w"></span>
<span class="w"> </span><span class="nx">contentsHooks</span><span class="w"> </span><span class="p">[]</span><span class="nx">Htmlizer</span><span class="w"></span>
<span class="w"> </span><span class="nx">pluginClients</span><span class="w"> </span><span class="p">[]</span><span class="o">*</span><span class="nx">goplugin</span><span class="p">.</span><span class="nx">Client</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>And the relevant part from its <tt class="docutils literal">LoadPlugins</tt> method:</p>
<div class="highlight"><pre><span></span><span class="c1">// Query the plugin for its capabilities -- the hooks it supports.</span><span class="w"></span>
<span class="c1">// Based on this information, register the plugin with the appropriate</span><span class="w"></span>
<span class="c1">// role or contents hooks.</span><span class="w"></span>
<span class="nx">capabilities</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">impl</span><span class="p">.</span><span class="nx">Hooks</span><span class="p">()</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">cap</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">range</span><span class="w"> </span><span class="nx">capabilities</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">cap</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"contents"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">contentsHooks</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nb">append</span><span class="p">(</span><span class="nx">m</span><span class="p">.</span><span class="nx">contentsHooks</span><span class="p">,</span><span class="w"> </span><span class="nx">impl</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">parts</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">strings</span><span class="p">.</span><span class="nx">Split</span><span class="p">(</span><span class="nx">cap</span><span class="p">,</span><span class="w"> </span><span class="s">":"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">len</span><span class="p">(</span><span class="nx">parts</span><span class="p">)</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s">"role"</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">m</span><span class="p">.</span><span class="nx">roleHooks</span><span class="p">[</span><span class="nx">parts</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">impl</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>It queries each plugins for its supported hooks, and then registers the right
hooks. As a result, when we encounter a role like <tt class="docutils literal">:tt:</tt>, it will only invoke
the plugin that asked to handle this role.</p>
<p>In this code <tt class="docutils literal">impl</tt> refers to a value of the type <tt class="docutils literal">PluginClientRPC</tt>, which
implements the <tt class="docutils literal">Htmlize</tt> interface by issuing RPC calls to the plugin:</p>
<div class="highlight"><pre><span></span><span class="c1">// PluginClientRPC is used by clients (main application) to translate the</span><span class="w"></span>
<span class="c1">// Htmlize interface of plugins to RPC calls.</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">PluginClientRPC</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="o">*</span><span class="nx">rpc</span><span class="p">.</span><span class="nx">Client</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">PluginClientRPC</span><span class="p">)</span><span class="w"> </span><span class="nx">Hooks</span><span class="p">()</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">reply</span><span class="w"> </span><span class="nx">HooksReply</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nx">Call</span><span class="p">(</span><span class="s">"Plugin.Hooks"</span><span class="p">,</span><span class="w"> </span><span class="nx">HooksArgs</span><span class="p">{},</span><span class="w"> </span><span class="o">&</span><span class="nx">reply</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">reply</span><span class="p">.</span><span class="nx">Hooks</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">PluginClientRPC</span><span class="p">)</span><span class="w"> </span><span class="nx">ProcessContents</span><span class="p">(</span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">reply</span><span class="w"> </span><span class="nx">ContentsReply</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nx">Call</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"Plugin.ProcessContents"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">ContentsArgs</span><span class="p">{</span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">Post</span><span class="p">:</span><span class="w"> </span><span class="nx">post</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="nx">reply</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">reply</span><span class="p">.</span><span class="nx">Value</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">c</span><span class="w"> </span><span class="o">*</span><span class="nx">PluginClientRPC</span><span class="p">)</span><span class="w"> </span><span class="nx">ProcessRole</span><span class="p">(</span><span class="nx">role</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">reply</span><span class="w"> </span><span class="nx">RoleReply</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">c</span><span class="p">.</span><span class="nx">client</span><span class="p">.</span><span class="nx">Call</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"Plugin.ProcessRole"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">RoleArgs</span><span class="p">{</span><span class="nx">Role</span><span class="p">:</span><span class="w"> </span><span class="nx">role</span><span class="p">,</span><span class="w"> </span><span class="nx">Value</span><span class="p">:</span><span class="w"> </span><span class="nx">val</span><span class="p">,</span><span class="w"> </span><span class="nx">Post</span><span class="p">:</span><span class="w"> </span><span class="nx">post</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="nx">reply</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">reply</span><span class="p">.</span><span class="nx">Value</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Per convention, each RPC call has its own type for arguments and another for
the response (e.g. <tt class="docutils literal">RoleArgs</tt> and <tt class="docutils literal">RoleReply</tt>). These are basic data
containers I'm leaving out of this post but you can see in the code.
A similar RPC wrapper is implemented on the server (plugin) side, doing the
translation the other way.</p>
<p><strong>Exposing application capabilities back to plugins:</strong> this is actually quite
tricky to accomplish in RPC-based systems, at least in the general case. Since
in this model plugins run in a separate process, we can't just pass a reference
to a big data structure into the plugin as we did before.</p>
<p>For small and simple data structures, serializing them through the RPC is not
an issue. This is what our example does for the <tt class="docutils literal">content.Post</tt> type - as you
can see from the <tt class="docutils literal">Htmlizer</tt> interface code snippet above. But what about
larger types? What if we want to expose the entire DB to the plugin? Or have the
plugin invoke functionality in the main application.</p>
<p>This is one of the capabilities <tt class="docutils literal"><span class="pre">go-plugin</span></tt> provides, via its <em>bidirectional
communication</em> feature. <tt class="docutils literal"><span class="pre">go-plugin</span></tt> can multiplex several RPC channels onto
the same connection between the plugin and the application (using the <a class="reference external" href="https://github.com/hashicorp/yamux">yamux
package</a>), and through this the client
can open its own RPC server available to the plugin to invoke.</p>
<p>I left this out of our example because I didn't want to needlessly complicate
it (we don't really need this functionality for <tt class="docutils literal">htmlize</tt>), but I created a
<a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-plugin-bidir-netrpc">separate sample</a>
that shows how to do this with <tt class="docutils literal">net/rpc</tt> - it's on GitHub if you're interested
<a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>.</p>
</div>
<div class="section" id="a-sample-plugin-for-htmlize">
<h2>A sample plugin for <tt class="docutils literal">htmlize</tt></h2>
<p>Here's the entire code for a sample plugin - one that implements rendering the
<tt class="docutils literal">:tt:</tt> role into the <tt class="docutils literal"><tt></tt> HTML element:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"fmt"</span><span class="w"></span>
<span class="w"> </span><span class="s">"example.com/content"</span><span class="w"></span>
<span class="w"> </span><span class="s">"example.com/plugin"</span><span class="w"></span>
<span class="w"> </span><span class="nx">goplugin</span><span class="w"> </span><span class="s">"github.com/hashicorp/go-plugin"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">TtHtmlizer</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">TtHtmlizer</span><span class="p">)</span><span class="w"> </span><span class="nx">Hooks</span><span class="p">()</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"role:tt"</span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">TtHtmlizer</span><span class="p">)</span><span class="w"> </span><span class="nx">ProcessContents</span><span class="p">(</span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">val</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">TtHtmlizer</span><span class="p">)</span><span class="w"> </span><span class="nx">ProcessRole</span><span class="p">(</span><span class="nx">role</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="w"> </span><span class="kt">string</span><span class="p">,</span><span class="w"> </span><span class="nx">post</span><span class="w"> </span><span class="nx">content</span><span class="p">.</span><span class="nx">Post</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Sprintf</span><span class="p">(</span><span class="s">"<tt>%s</tt>"</span><span class="p">,</span><span class="w"> </span><span class="nx">val</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">goplugin</span><span class="p">.</span><span class="nx">Serve</span><span class="p">(</span><span class="o">&</span><span class="nx">goplugin</span><span class="p">.</span><span class="nx">ServeConfig</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">HandshakeConfig</span><span class="p">:</span><span class="w"> </span><span class="nx">plugin</span><span class="p">.</span><span class="nx">Handshake</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Plugins</span><span class="p">:</span><span class="w"> </span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="nx">goplugin</span><span class="p">.</span><span class="nx">Plugin</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"htmlize"</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">plugin</span><span class="p">.</span><span class="nx">HtmlizePlugin</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Impl</span><span class="p">:</span><span class="w"> </span><span class="nx">TtHtmlizer</span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Using <tt class="docutils literal"><span class="pre">go-plugin</span></tt> involves a bit of work for setting up the RPC scaffolding,
but once that's all done, writing new plugins is quick and easy: just implement
an interface and invoke a plugin server registration function in <tt class="docutils literal">main</tt>.</p>
</div>
<div class="section" id="comparing-shared-libraries-vs-rpc-for-plugins">
<h2>Comparing shared libraries vs RPC for plugins</h2>
<p>By now we've seen how to create run-time plugins in Go using two different
mechanisms: shared libraries (<tt class="docutils literal"><span class="pre">--buildmode=plugin</span></tt>) and RPC (via
<tt class="docutils literal"><span class="pre">go-plugin</span></tt>). Which one should you actually use?</p>
<p>As the <a class="reference external" href="https://eli.thegreenplace.net/2021/plugins-in-go/">previous post</a>
discussed, shared library plugins can be awkward to work with, because they
require strict source code compatibility with the main application. Another
problem is portability: they don't currently work on Windows, for example, and
it's not clear if they ever will.</p>
<p>On the other hand, shared library plugins have excellent performance: plugins
run in-process with the main application. Calling a plugin is a simple Go
function call and we can pass references to data structures across the boundary.
Compare that to RPC, where each call is a network interaction; even if this
"network" is very fast (Unix domain sockets or localhost TCP), it still implies
serializing the data into a linear buffer, then deserializing it at the other
end; this is orders of magnitude slower per invocation.</p>
<p>RPC-based plugins have some clear benefits of their own, however. First and
foremost - isolation; a plugin runs in a process of its own. If it crashes,
it doesn't bring down the rest of the application with it.
Moreover, if the plugin is malicious there's limited damage it can do to the
application itself. Plugins can even be invoked with different system
permissions from the main application. With some tinkering it should be possible
to set up a plugin that runs inside a container.</p>
<p>An additional benefit is that plugins can be distributed; since they can talk
RPC over TCP. It's easy to set up a plugin that runs on a different machine
(<tt class="docutils literal"><span class="pre">go-plugin</span></tt> supports this with its <tt class="docutils literal">ReattachConfig</tt> type).</p>
<p>Finally, since the interface is RPC, the plugins don't even necessarily have to
be written in Go! If we use the gRPC interface, we can theoretically use any
language to write a plugin for a Go application; in this scenario, all the data
exchanged between the main application and plugins goes through protocol buffers
anyway.</p>
</div>
<div class="section" id="source-code">
<h2>Source code</h2>
<p>The full source code for this sample is here: <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-plugin-htmlize-rpc">go-plugin-htmlize-rpc</a>.</p>
<p>Separate sample that shows how to call back from plugins into the host with
<tt class="docutils literal"><span class="pre">go-plugin</span></tt> and <tt class="docutils literal">net/rpc</tt>: <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2023/go-plugin-bidir-netrpc">go-plugin-bidir-netrpc</a>.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>To be precise, it's RPC scaffolding implemented by us + <tt class="docutils literal"><span class="pre">go-plugin</span></tt>
that do this in tandem. As described earlier, in order to support
multiple RPC flavors, <tt class="docutils literal"><span class="pre">go-plugin</span></tt> leaves the RPC layer scaffolding
for users to define.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>The <tt class="docutils literal"><span class="pre">go-plugin</span></tt> repository has an example of doing this for gRPC
but not for <tt class="docutils literal">net/rpc</tt>, as far as I could tell.</td></tr>
</tbody>
</table>
</div>
Reverse proxying a sub-domain via Apache2023-01-21T06:26:00-08:002023-01-21T14:26:05-08:00Eli Benderskytag:eli.thegreenplace.net,2023-01-21:/2023/reverse-proxying-a-sub-domain-via-apache/<p>Suppose you have a domain that hosts your website: <tt class="docutils literal">domain.com</tt>, and the
website is served with the venerable <a class="reference external" href="https://en.wikipedia.org/wiki/Apache_HTTP_Server">Apache HTTP server</a>. Suppose, also, that you
want to run some backend application on the same domain, perhaps using a
sub-domain like <tt class="docutils literal">sub.domain.com</tt>. Running an application on a non-standard …</p><p>Suppose you have a domain that hosts your website: <tt class="docutils literal">domain.com</tt>, and the
website is served with the venerable <a class="reference external" href="https://en.wikipedia.org/wiki/Apache_HTTP_Server">Apache HTTP server</a>. Suppose, also, that you
want to run some backend application on the same domain, perhaps using a
sub-domain like <tt class="docutils literal">sub.domain.com</tt>. Running an application on a non-standard
port (not 80 or 443) is not a problem, but what it you need it to run on port
80? Apache occupies port 80 in order to serve <tt class="docutils literal">domain.com</tt>, so at least on
the surface this seems like a problem.</p>
<img alt="Logo of the Apache HTTP server project" class="align-center" src="https://eli.thegreenplace.net/images/2023/apache-logo.png" style="width: 500px;" />
<p>This post talks about how to make it work using the reverse-proxying
capabilities of Apache. It assumes you control a virtual machine that has
a top-level domain like <tt class="docutils literal">domain.com</tt> mapped to it, and that the machine runs
Linux.</p>
<div class="section" id="setting-up-apache-as-a-proxy-with-mod-proxy">
<h2>Setting up Apache as a proxy with mod_proxy</h2>
<p>If you need to brush up on proxy concepts, consider reading <a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">this series of
posts</a>
first.</p>
<p>Assuming Apache is already installed and running on the server, you'll first
have to enable the <a class="reference external" href="https://httpd.apache.org/docs/2.4/mod/mod_proxy.html">proxy module</a> and restart the
service:</p>
<div class="highlight"><pre><span></span>$ sudo a2enmod proxy proxy_http
$ sudo systemctl restart apache2
</pre></div>
<p>Sub-domains typically have their own configuration file in
<tt class="docutils literal"><span class="pre">/etc/apache2/sites-available</span></tt>. Create a new configuration file in
that directory, named <tt class="docutils literal">sub.domain.com.conf</tt> or some such; here's what should
be in it (adjust as needed):</p>
<div class="highlight"><pre><span></span><VirtualHost *:80>
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:5000/
ProxyPassReverse / http://127.0.0.1:5000/
ServerName sub.domain.com
ServerAdmin your@email.com
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
</pre></div>
<p>This tells Apache that the <tt class="docutils literal">sub.domain.com</tt> route should be proxied to a
service running locally on port 5000; naturally, the service address can have a
different port or run on a different domain altogether.</p>
<p>Next you'll want to register that configuration with Apache and restart it
again:</p>
<div class="highlight"><pre><span></span>$ sudo a2ensite sub.domain.com.conf
$ sudo systemctl restart apache2
</pre></div>
</div>
<div class="section" id="running-the-backend-service">
<h2>Running the backend service</h2>
<p>Now that Apache is all set up, it's time to run the actual backend service at
port 5000. As an example, you can run
<a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/http-server-debug-request-headers.go">this simple header debugging server</a>:</p>
<div class="highlight"><pre><span></span>$ go run http-server-debug-request-headers.go -addr 127.0.0.1:5000
2023/01/17 01:01:20 Starting server on 127.0.0.1:5000
</pre></div>
<p>To test that it runs properly, in a separate terminal (on the same machine!)
let's run <tt class="docutils literal">curl</tt>:</p>
<div class="highlight"><pre><span></span>$ curl 127.0.0.1:5000/headers
hello /headers
</pre></div>
<p>And looking at the terminal where the server is running, you should see some
useful logging:</p>
<div class="highlight"><pre><span></span>2023/01/17 01:02:50 127.0.0.1:42406 GET /headers Host: 127.0.0.1:5000
User-Agent: curl/7.81.0
Accept: */*
</pre></div>
<p>If you've followed all the steps in this and the previous session, it should
work via the sub-domain now (from any machine):</p>
<div class="highlight"><pre><span></span>$ curl http://sub.domain.com/headers
hello /headers
</pre></div>
<p>Apache listens on port 80 for <tt class="docutils literal">domain.com</tt>, and when it sees requests to
<tt class="docutils literal">sub.domain.com</tt>, it proxies them to the server running on port <tt class="docutils literal">5000</tt> on
the same machine.</p>
<p>If this doesn't work for you, take a careful look at the Apache logs - both
the error log and the access log may be useful.</p>
</div>
<div class="section" id="bonus-tls-with-let-s-encrypt">
<h2>Bonus: TLS with Let's Encrypt</h2>
<p>If your server is set up to serve <tt class="docutils literal">domain.com</tt> via TLS using Let's Encrypt,
I have good news for you -- it will <em>just work</em> for <tt class="docutils literal">sub.domain.com</tt> as well!</p>
<p>Presumably you've set up Let's Encrypt certificates using <tt class="docutils literal">certbot</tt>. Since
we've now added an additional Apache configuration (<tt class="docutils literal">sub.domain.com.conf</tt>), we
should run <tt class="docutils literal">certbot</tt> again:</p>
<div class="highlight"><pre><span></span>$ sudo certbot --apache
</pre></div>
<p>And carefully follow the on-screen instructions. <tt class="docutils literal">certbot</tt> should detect
there's a new sub-domain to get a certificate for; if everything goes as
expected, it succeeds and from that point on you should be able to access the
backend server via HTTPS:</p>
<div class="highlight"><pre><span></span>$ curl https://sub.domain.com/headers
hello /headers
</pre></div>
<p>Note that the backend Go server serves HTTP; the reverse proxy (Apache)
terminates the TLS connection and passes HTTP to the backend server. This is
a fairly common way to structure backends. While the backend server
serves unencrypted traffic, it's not actually accessible from outside the
machine (port 5000 is unlikely to be exposed). The only way to access it is
via the reverse-proxy on <tt class="docutils literal">sub.domain.com</tt>, which can use TLS if needed.</p>
<p>I was wondering how this works. <tt class="docutils literal">certbot</tt> uses the HTTP challenge with Let's
Encrypt, wherein it's asked to serve a special file on a special path
(typically something like <tt class="docutils literal"><span class="pre">.well-known/acme-challenge</span></tt>) to prove to Let's
Encrypt that it controls the domain. But here all requests get forwarded to
the backend server...</p>
<p>After scratching my head for a minute I found the answer in <tt class="docutils literal">certbot</tt>'s logs,
where it honestly explains its tricky ways. It turns out it adds a
<tt class="docutils literal">RewriteRule</tt> to our <tt class="docutils literal">sub.domain.com.conf</tt> file for the duration of the
Let's Encrypt handshake, sending any requests starting with
<tt class="docutils literal"><span class="pre">.well-known/acme-challenge</span></tt> to a known disk location it controls. After all
is done, it quietly removes these rules from the configuration file.</p>
</div>
Go and Proxy Servers: Part 3 - SOCKS proxies2022-12-14T20:10:00-08:002022-12-15T13:27:28-08:00Eli Benderskytag:eli.thegreenplace.net,2022-12-14:/2022/go-and-proxy-servers-part-3-socks-proxies/<p>This is the third post in a series about proxy servers and Go. Here is a list of
posts in the series:</p>
<ul class="simple">
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">Part 1 - HTTP Proxies</a></li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-2-https-proxies/">Part 2 - HTTPS Proxies</a></li>
<li>Part 3 - SOCKS Proxies (this part)</li>
</ul>
<p>So far the series has covered HTTP and HTTPS proxies; this part is a …</p><p>This is the third post in a series about proxy servers and Go. Here is a list of
posts in the series:</p>
<ul class="simple">
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">Part 1 - HTTP Proxies</a></li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-2-https-proxies/">Part 2 - HTTPS Proxies</a></li>
<li>Part 3 - SOCKS Proxies (this part)</li>
</ul>
<p>So far the series has covered HTTP and HTTPS proxies; this part is a brief
discussion of SOCKS - an old and venerable proxy protocol designed to relay any
kind of TCP or UDP traffic through forward proxies.</p>
<div class="section" id="a-bit-of-history">
<h2>A bit of history</h2>
<p>The SOCKS protocol has been around for a while; the latest version and the one
you care about in 2022 is SOCKS5, which was specified by <a class="reference external" href="https://datatracker.ietf.org/doc/html/rfc1928">RFC 1928</a> back in 1996. The motivation
for its design back then wasn't too different from what we're using forward
proxies for today - dialing through firewalls. It predates HTTP version 1.1 and
the <tt class="docutils literal">CONNECT</tt> method, which is important context to keep in mind while reading
this post. A brief comparison of SOCKS5 and <tt class="docutils literal">CONNECT</tt> proxies is included
later in the post.</p>
</div>
<div class="section" id="dialing-through-a-socks5-server-in-go">
<h2>Dialing through a SOCKS5 server in Go</h2>
<p>This post won't explain the SOCKS5 protocol in detail; please read the RFC for
that - it's short and quite readable. We'll move straight ahead to basic usage
in Go - having clients dial through a SOCKS5 proxy. We'll focus on HTTP here,
though SOCKS5 supports any TCP or UDP traffic.</p>
<p>SOCKS5 is used for <em>forward</em> proxies (to recall what this means, check out <a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">part
1</a>
in this series), and the setup is similar to other proxies. There's a proxy
server running on some host and port, and when accessing web pages we have to
tell our client to use the proxy. The traditional port for SOCKS5 is 1080.</p>
<p>To set up a demonstration, I'm using <a class="reference external" href="https://github.com/rofl0r/microsocks">microsocks</a> - a small SOCKS5 implementation (in C)
I found on GitHub; it's pretty easy to clone and build locally. Once that's
done, invoke it as follows:</p>
<div class="highlight"><pre><span></span>$ ./microsocks
</pre></div>
<p>This runs the proxy service listening on port 1080 without authentication; I'll
have more to say about authentication later.</p>
<p>We can now <tt class="docutils literal">curl</tt> through this proxy:</p>
<div class="highlight"><pre><span></span>$ http_proxy=socks5://localhost:1080 curl -v http://example.org
* Uses proxy env variable http_proxy == 'socks5://localhost:1080'
* Trying 127.0.0.1:1080...
* SOCKS5 connect to IPv4 93.184.216.34:80 (locally resolved)
* SOCKS5 request granted.
* Connected to (nil) (127.0.0.1) port 1080 (#0)
> GET / HTTP/1.1
> Host: example.org
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
// ... rest of response
</pre></div>
<p>And if you look at the terminal where <tt class="docutils literal">microsocks</tt> is running you should see
a logging message confirming that a client was connected.</p>
<p>The "magic" here is done with the <tt class="docutils literal">http_proxy</tt> environment variable, using the
<tt class="docutils literal"><span class="pre">socks5://</span></tt> protocol prefix for the proxy address. The same works by default
for Go's <tt class="docutils literal">net/http</tt> - we can similarly run our our <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/http-get-basic.go">simple HTTP client</a>:</p>
<div class="highlight"><pre><span></span>$ http_proxy=socks5://localhost:1080 go run http-get-basic.go http://example.org
Response status: 200 OK
<!doctype html>
// ... rest of response
</pre></div>
<p>Naturally, the same approach works for HTTPS, using the <tt class="docutils literal">https_proxy</tt> env
var:</p>
<div class="highlight"><pre><span></span>$ https_proxy=socks5://localhost:1080 curl -v https://example.org
* Uses proxy env variable https_proxy == 'socks5://localhost:1080'
* Trying 127.0.0.1:1080...
* SOCKS5 connect to IPv4 93.184.216.34:443 (locally resolved)
* SOCKS5 request granted.
* Connected to (nil) (127.0.0.1) port 1080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
// ... rest of TLS handshake and response
</pre></div>
<p>Just like a <a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-2-https-proxies/">CONNECT tunnel</a>,
SOCKS5 is oblivious to the contents of the traffic flowing through it once the
forwarding is set up, treating it as a stream of opaque bytes; it
works well with any protocol streamed through it - including TLS.</p>
</div>
<div class="section" id="socks5-authentication">
<h2>SOCKS5 authentication</h2>
<p>The SOCKS5 proxy protocol supports <em>authentication</em> using several methods, to
ensure that only authorized clients can access the proxy. The authentication
part of the protocol was designed to be extensible, but we'll just focus on
the basic method described in the original RFC: username/password.</p>
<p>We can run our <tt class="docutils literal">microsocks</tt> proxy again, this time protected by a username
and password:</p>
<div class="highlight"><pre><span></span>$ ./microsocks -u myuser -P mypass
</pre></div>
<p>First, let's try to <tt class="docutils literal">curl</tt> through this proxy without setting authentication:</p>
<div class="highlight"><pre><span></span>$ http_proxy=socks5://localhost:1080 curl -v http://example.org
* Uses proxy env variable http_proxy == 'socks5://localhost:1080'
* Trying 127.0.0.1:1080...
* No authentication method was acceptable.
* Closing connection 0
curl: (97) No authentication method was acceptable.
</pre></div>
<p>Now let's actually set matching credentials in the proxy URL (note the username
and password preceding the address):</p>
<div class="highlight"><pre><span></span>$ http_proxy=socks5://myuser:mypass@localhost:1080 curl -v http://example.org
* Uses proxy env variable http_proxy == 'socks5://myuser:mypass@localhost:1080'
* Trying 127.0.0.1:1080...
* SOCKS5 connect to IPv4 93.184.216.34:80 (locally resolved)
* SOCKS5 request granted.
* Connected to (nil) (127.0.0.1) port 1080 (#0)
> GET / HTTP/1.1
> Host: example.org
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
// ... rest of response
</pre></div>
<p>And the same setting will work with Go's HTTP client by default. If we don't
want to require setting environment variables, we can also do this
programmatically in Go as follows:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"flag"</span><span class="w"></span>
<span class="w"> </span><span class="s">"fmt"</span><span class="w"></span>
<span class="w"> </span><span class="s">"io/ioutil"</span><span class="w"></span>
<span class="w"> </span><span class="s">"log"</span><span class="w"></span>
<span class="w"> </span><span class="s">"net/http"</span><span class="w"></span>
<span class="w"> </span><span class="s">"golang.org/x/net/proxy"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">target</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"target"</span><span class="p">,</span><span class="w"> </span><span class="s">"http://example.org"</span><span class="p">,</span><span class="w"> </span><span class="s">"URL to get"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">proxyAddr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"proxy"</span><span class="p">,</span><span class="w"> </span><span class="s">"localhost:1080"</span><span class="p">,</span><span class="w"> </span><span class="s">"SOCKS5 proxy address to use"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">username</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"user"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"username for SOCKS5 proxy"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"pass"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"password for SOCKS5 proxy"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">auth</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">proxy</span><span class="p">.</span><span class="nx">Auth</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">User</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">username</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Password</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">password</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">dialer</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">proxy</span><span class="p">.</span><span class="nx">SOCKS5</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">proxyAddr</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="nx">auth</span><span class="p">,</span><span class="w"> </span><span class="kc">nil</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">client</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">http</span><span class="p">.</span><span class="nx">Client</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Transport</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">http</span><span class="p">.</span><span class="nx">Transport</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Dial</span><span class="p">:</span><span class="w"> </span><span class="nx">dialer</span><span class="p">.</span><span class="nx">Dial</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">r</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Get</span><span class="p">(</span><span class="o">*</span><span class="nx">target</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">r</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">body</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ioutil</span><span class="p">.</span><span class="nx">ReadAll</span><span class="p">(</span><span class="nx">r</span><span class="p">.</span><span class="nx">Body</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="nb">string</span><span class="p">(</span><span class="nx">body</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>The <tt class="docutils literal">golang.org/x/net/proxy</tt> package provides explicit tools for proxies,
and here specifically we're using its <tt class="docutils literal">SOCKS5</tt> dialer as a custom <tt class="docutils literal">Dial</tt>
in a <tt class="docutils literal">Transport</tt>. We can run this client as follows:</p>
<div class="highlight"><pre><span></span>$ go run http-get-socks-transport.go -proxy localhost:1080 \
-user myuser -pass mypass \
http://example.org
<!doctype html>
<html>
// ... rest of response
</pre></div>
</div>
<div class="section" id="socks5-vs-connect-tunnels">
<h2>SOCKS5 vs. CONNECT tunnels</h2>
<p>In <a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-2-https-proxies/">part 2 of this series</a>
we've covered proxying arbitrary traffic via a standard HTTP server by means of
the <tt class="docutils literal">CONNECT</tt> method. What is the difference between <tt class="docutils literal">CONNECT</tt>-based proxies
and SOCKS5 proxies?</p>
<p>First, to repeat a bit of historic context: the SOCKS protocol predates the
<tt class="docutils literal">CONNECT</tt> method. It requires a dedicated service listening on a dedicated
port, whereas handling <tt class="docutils literal">CONNECT</tt> tunnels can be included in an existing HTTP
server and share the port.</p>
<p>On the other hand, while SOCKS5 can handle any TCP or UDP traffic, <tt class="docutils literal">CONNECT</tt>
handles only TCP <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
<p>Finally, note that the security aspects of SOCKS5 are rather primitive, whereas
HTTP proxies can use TLS under the hood to handle both client-server
authentication and traffic encryption. It is fair to mention, however, that it's
not too hard to combine protocols and "wrap" SOCKS5 in TLS - in fact, this is
what <tt class="docutils literal">ssh <span class="pre">-D</span></tt> does already. An alternative would be to run SOCKS5 via <a class="reference external" href="https://eli.thegreenplace.net/2022/ssh-port-forwarding-with-go/">SSH
port forwarding</a>.</p>
</div>
<div class="section" id="socks5-server-in-go">
<h2>SOCKS5 server in Go</h2>
<p>There are a number of open-source SOCKS5 written in Go; one I liked for its
clarity and simplicity is <a class="reference external" href="https://github.com/armon/go-socks5">go-socks5</a>,
a project from one of Hashicorp's co-founders. A basic sample of setting up
a SOCKS5 server using this package is available in its README. I will show a
slightly more advanced sample that uses basic authentication just like the
microsocks example discussed earlier:</p>
<div class="highlight"><pre><span></span><span class="kn">package</span><span class="w"> </span><span class="nx">main</span><span class="w"></span>
<span class="kn">import</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="s">"flag"</span><span class="w"></span>
<span class="w"> </span><span class="s">"github.com/armon/go-socks5"</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">myCredentialStore</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">user</span><span class="w"> </span><span class="kt">string</span><span class="w"></span>
<span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="kt">string</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">cs</span><span class="w"> </span><span class="o">*</span><span class="nx">myCredentialStore</span><span class="p">)</span><span class="w"> </span><span class="nx">Valid</span><span class="p">(</span><span class="nx">user</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">user</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">cs</span><span class="p">.</span><span class="nx">user</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">cs</span><span class="p">.</span><span class="nx">password</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">username</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"u"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"username for SOCKS5 proxy"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">password</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"P"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"password for SOCKS5 proxy"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">auth</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">socks5</span><span class="p">.</span><span class="nx">UserPassAuthenticator</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Credentials</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">myCredentialStore</span><span class="p">{</span><span class="nx">user</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">password</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">password</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">conf</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">socks5</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">AuthMethods</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nx">socks5</span><span class="p">.</span><span class="nx">Authenticator</span><span class="p">{</span><span class="nx">auth</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">server</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">socks5</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="nx">conf</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">server</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="s">"127.0.0.1:1080"</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">panic</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>We can run this server as follows:</p>
<div class="highlight"><pre><span></span>$ go run . -u myuser -P mypass
</pre></div>
<p>And then <tt class="docutils literal">curl</tt> as before:</p>
<div class="highlight"><pre><span></span>$ http_proxy=socks5://myuser:mypass@localhost:1080 curl -v http://example.org
* Uses proxy env variable http_proxy == 'socks5://myuser:mypass@localhost:1080'
* Trying 127.0.0.1:1080...
* SOCKS5 connect to IPv4 93.184.216.34:80 (locally resolved)
* SOCKS5 request granted.
* Connected to (nil) (127.0.0.1) port 1080 (#0)
> GET / HTTP/1.1
> Host: example.org
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
// ... rest of response
</pre></div>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>It should be noted that many open-source SOCKS5 servers do not, in fact,
implement UDP since it's not used often. For example, the two servers
discussed in this post - microsocks and go-socks5 - have no UDP support.</td></tr>
</tbody>
</table>
</div>
SSH port forwarding with Go2022-11-19T19:22:00-08:002022-11-20T03:25:48-08:00Eli Benderskytag:eli.thegreenplace.net,2022-11-19:/2022/ssh-port-forwarding-with-go/<p>This post shows how to set up SSH port forwarding ("tunnels") - both local and
remote - using the extended Go standard library.</p>
<div class="section" id="setup">
<h2>Setup</h2>
<p>While you <em>could</em> set up <tt class="docutils literal">localhost</tt> forwarding for testing, to discuss a more
realistic scenario I would recommend spinning up a basic <a class="reference external" href="https://en.wikipedia.org/wiki/Virtual_private_server">VPS</a>. For the purpose of
writing …</p></div><p>This post shows how to set up SSH port forwarding ("tunnels") - both local and
remote - using the extended Go standard library.</p>
<div class="section" id="setup">
<h2>Setup</h2>
<p>While you <em>could</em> set up <tt class="docutils literal">localhost</tt> forwarding for testing, to discuss a more
realistic scenario I would recommend spinning up a basic <a class="reference external" href="https://en.wikipedia.org/wiki/Virtual_private_server">VPS</a>. For the purpose of
writing this post, I run a bare-bones Ubuntu VPS on Digital Ocean with the
public IP address
159.89.238.232 (at the time of writing) and a <tt class="docutils literal">root</tt> user. You can easily do
the same using any cloud provider (obviously, accessing <em>my</em> VPS won't work
for you since it requires SSH authentication with a known set of keys).</p>
</div>
<div class="section" id="testing-the-setup-basic-remote-command-execution">
<h2>Testing the setup - basic remote command execution</h2>
<p>Before we start talking about tunnels, let's test our setup by doing something
much simpler - basic SSH remote command execution. Here's the equivalent
command-line invocation:</p>
<div class="highlight"><pre><span></span>$ ssh root@159.89.238.232 'uname -a'
Linux testdrop6 5.19.0-23-generic #24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 05:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
</pre></div>
<p>This runs the command <tt class="docutils literal">uname <span class="pre">-a</span></tt> on my VPS. Here are a few things to be aware
of:</p>
<ul class="simple">
<li>The <tt class="docutils literal">sshd</tt> server should be running on the remote server (VPS); run
<tt class="docutils literal">service ssh status</tt> to double check. I use OpenSSH's implementations of
both the <tt class="docutils literal">sshd</tt> server and <tt class="docutils literal">ssh</tt> client, but presumably other
implementations could work as well.</li>
<li>The default SSH port (22) should be open in whatever firewall the remote
server is running. If your VPS runs Ubuntu, check the output of
<tt class="docutils literal">ufw status</tt>.</li>
<li>The first time you connect to the VPS with <tt class="docutils literal">ssh</tt>, it will ask you about
checking the server's host key. Either do the due diligence to verify the key
or just blindly accept it, but it's important for the VPS to have an entry
in your local <tt class="docutils literal">known_hosts</tt> file.</li>
</ul>
<p>Now let's see how to accomplish the same task in Go; we'll be using "extended
stdlib" package <a class="reference external" href="https://pkg.go.dev/golang.org/x/crypto/ssh">https://pkg.go.dev/golang.org/x/crypto/ssh</a> for this purpose. The
full code for the samples in this post is <a class="reference external" href="https://github.com/eliben/code-for-blog/tree/master/2022/go-ssh">available on GitHub</a>.</p>
<p>We'll start with the code that sets up the SSH client configuration for us:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">createSshConfig</span><span class="p">(</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="nx">keyFile</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="nx">ssh</span><span class="p">.</span><span class="nx">ClientConfig</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">knownHostsCallback</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">knownhosts</span><span class="p">.</span><span class="nx">New</span><span class="p">(</span><span class="nx">sshConfigPath</span><span class="p">(</span><span class="s">"known_hosts"</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">ReadFile</span><span class="p">(</span><span class="nx">keyFile</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatalf</span><span class="p">(</span><span class="s">"unable to read private key: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Create the Signer for this private key.</span><span class="w"></span>
<span class="w"> </span><span class="nx">signer</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ssh</span><span class="p">.</span><span class="nx">ParsePrivateKey</span><span class="p">(</span><span class="nx">key</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatalf</span><span class="p">(</span><span class="s">"unable to parse private key: %v"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// An SSH client is represented with a ClientConn.</span><span class="w"></span>
<span class="w"> </span><span class="c1">//</span><span class="w"></span>
<span class="w"> </span><span class="c1">// To authenticate with the remote server you must pass at least one</span><span class="w"></span>
<span class="w"> </span><span class="c1">// implementation of AuthMethod via the Auth field in ClientConfig,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and provide a HostKeyCallback.</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="o">&</span><span class="nx">ssh</span><span class="p">.</span><span class="nx">ClientConfig</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">User</span><span class="p">:</span><span class="w"> </span><span class="nx">username</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Auth</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nx">ssh</span><span class="p">.</span><span class="nx">AuthMethod</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">ssh</span><span class="p">.</span><span class="nx">PublicKeys</span><span class="p">(</span><span class="nx">signer</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nx">HostKeyCallback</span><span class="p">:</span><span class="w"> </span><span class="nx">knownHostsCallback</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">HostKeyAlgorithms</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="nx">ssh</span><span class="p">.</span><span class="nx">KeyAlgoED25519</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">sshConfigPath</span><span class="p">(</span><span class="nx">filename</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">filepath</span><span class="p">.</span><span class="nx">Join</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nx">Getenv</span><span class="p">(</span><span class="s">"HOME"</span><span class="p">),</span><span class="w"> </span><span class="s">".ssh"</span><span class="p">,</span><span class="w"> </span><span class="nx">filename</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>In the interest of brevity, this code is not particularly generic: it's somewhat
tuned for my local setup. It takes the VPS username and the path to the
local private SSH key as parameters.</p>
<p>First, it configures a "host callback" which is the same host key validation
mechanism described earlier in the context of the <tt class="docutils literal">ssh</tt> command-line client.
The method I'm using in this sample is reading from the local <tt class="docutils literal">known_hosts</tt>
file, where the VPS will already be listed (since we accessed it with <tt class="docutils literal">ssh</tt>
earlier). This part of the setup is a bit finicky and if you're running into
trouble and just want to make progress, consider using
<tt class="docutils literal">ssh.InsecureIgnoreHostKey</tt> instead.</p>
<p>Next, it reads my private key and sets up the <tt class="docutils literal">ClientConfig</tt>. The <tt class="docutils literal">ED25519</tt>
public key algorithm is expected, since this is the one I'm using for my
SSH keys.</p>
<p>With a client configuration in hand, the rest of the code is straightforward:</p>
<div class="highlight"><pre><span></span><span class="nx">config</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">createSshConfig</span><span class="p">(</span><span class="o">*</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">keyFile</span><span class="p">)</span><span class="w"></span>
<span class="nx">client</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ssh</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">config</span><span class="p">)</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Failed to dial: "</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="c1">// Each ClientConn can support multiple interactive sessions,</span><span class="w"></span>
<span class="c1">// represented by a Session.</span><span class="w"></span>
<span class="nx">session</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">NewSession</span><span class="p">()</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Failed to create session: "</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">defer</span><span class="w"> </span><span class="nx">session</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="c1">// Once a Session is created, you can a single command on</span><span class="w"></span>
<span class="c1">// the remote side using the Run method.</span><span class="w"></span>
<span class="nx">session</span><span class="p">.</span><span class="nx">Stdout</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">os</span><span class="p">.</span><span class="nx">Stdout</span><span class="w"></span>
<span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">session</span><span class="p">.</span><span class="nx">Run</span><span class="p">(</span><span class="s">"uname -a"</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Failed to run: "</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>We create a client and then an interactive session (this is similar to the
interactive session we get if we simply <tt class="docutils literal">ssh</tt> into a server). Then the
session is used to <tt class="docutils literal">Run</tt> a command. Invoking this Go program we should get
the same output as the earlier <tt class="docutils literal">ssh</tt> command-line run:</p>
<div class="highlight"><pre><span></span>$ go run ssh-execute-remote-cmd.go \
-addr 159.89.238.232:22 -user root \
-keyfile ~/.ssh/id_ed25519
Linux testdrop6 5.19.0-23-generic #24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 05:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
</pre></div>
</div>
<div class="section" id="local-port-forwarding">
<h2>Local port forwarding</h2>
<p>Now that our setup is working, let's see how to implement a local port
forwarding server in Go. But first, what does <em>local port forwarding</em> mean?
The terminology can be a bit confusing here, so let's use a diagram. We'll start
with the components:</p>
<img alt="Basic diagram showing a local machine and remote machine with public IP" class="align-center" src="https://eli.thegreenplace.net/images/2022/local-remote-machine-basic.png" />
<p>The characters in this story are:</p>
<ol class="arabic simple">
<li>The "local machine" - your laptop, home desktop or whatever you're using to
read these posts and run the examples. Your local machine typically doesn't
have a public-facing IP address because it's behind layers of NAT, routers
and so on.</li>
<li>The "remote machine" - any cloud server / VPS you control, that has at least
a temporary IP address; just like the sample VPS I described earlier in this
post.</li>
</ol>
<p>Local port forwarding means that we use SSH to create a tunnel (a logical
bi-directional passthrough connection) between a port on the remote machine and
a port on the local machine, and forward connections from local to remote.
Here's the flow of events:</p>
<img alt="Local port forwarding flow of events" class="align-center" src="https://eli.thegreenplace.net/images/2022/local-port-forwarding.png" />
<p>It starts by having some server listen on port M on the remote machine. For
example, the PostgreSQL server listens on port 5432. This port is most likely
<em>not</em> exposed outside the machine for security and other reasons. But suppose we
want to talk with our PostgreSQL database on the remote machine using <tt class="docutils literal">psql</tt>
from our local machine.</p>
<p>In step 2, the <tt class="docutils literal">ssh</tt> client is used to establish local port forwarding from
port N on our local machine to port M on the remote machine. The <tt class="docutils literal">ssh</tt> client
contacts the <tt class="docutils literal">sshd</tt> server running on the VPS (typically over the standard SSH
port 22, but this can be configured) and they set up this connection.</p>
<p>From this point on, local port N gives us a "tunnel" to remote port M.
Connections to <tt class="docutils literal">localhost:N</tt> on the local machine will be automatically
connected to port <tt class="docutils literal">M</tt> on the remote machine. To be completely clear: data will
flow from the local client using port <tt class="docutils literal">N</tt> to the local <tt class="docutils literal">ssh</tt> client, from
there to the remote <tt class="docutils literal">sshd</tt> server which in turn forwards it to the remote port
<tt class="docutils literal">M</tt>; see the Appendix for more details.</p>
<p>Let's test this using the <tt class="docutils literal">ssh</tt> client and our VPS setup. I'll use one of my
favorite tools - netcat - to demonstrate how this works. First, I run this
on my VPS:</p>
<div class="highlight"><pre><span></span>remote# nc -lvk 7780
Listening on 0.0.0.0 7780
</pre></div>
<p>This creates a TCP server listening on port 7780 and echoing any data it gets
to stdout. Port 7780 is not exposed outside the VPS; it's not on the open ports
list of <tt class="docutils literal">ufw</tt>. I cannot connect to it directly from my local machine.</p>
<p>Let's set up local port forwarding now. Running this on the local machine:</p>
<div class="highlight"><pre><span></span>$ ssh -N -L 7777:localhost:7780 root@159.89.238.232
</pre></div>
<p>The syntax here is a bit funky (read <tt class="docutils literal">man ssh</tt> for the gory details), but in
a nutshell this means: create a tunnel between localhost port 7777 and port
7780 on the given remote machine. Note that we have to give <tt class="docutils literal">ssh</tt> the username
and IP address of the remote machine, just like when we were executing remote
commands <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
<p>Having run the above <tt class="docutils literal">ssh</tt> command, we can treat local port 7777 as if it's
a tunnel into the remote port 7780. <tt class="docutils literal">ssh</tt> listens on 7777 and forwards all
connections where needed. We can test this by running <tt class="docutils literal">nc</tt> again, this time
locally and in client mode:</p>
<div class="highlight"><pre><span></span>$ echo "foo bar" | nc -N localhost 7777
</pre></div>
<p>Here we instruct <tt class="docutils literal">nc</tt> to open a TCP connection to <tt class="docutils literal">localhost:7777</tt>, send the
message "foo bar" and close the connection. Looking at the server logs for our
remote machine we'll see:</p>
<div class="highlight"><pre><span></span>remote# nc -lvk 7780
Listening on 0.0.0.0 7780
Connection received on localhost 53090
foo bar
</pre></div>
<p>Now that we've seen how to set up a tunnel using the standard <tt class="docutils literal">ssh</tt> client,
let's turn our attention to Go. Here's a program to establish a local tunnel:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"addr"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"ssh server address to dial as <hostname>:<port>"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">username</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"user"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"username for ssh"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">keyFile</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"keyfile"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"file with private key for SSH authentication"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">remotePort</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"rport"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"remote port for tunnel"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">localPort</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"lport"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"local port for tunnel"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">createSshConfig</span><span class="p">(</span><span class="o">*</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">keyFile</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">client</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ssh</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">config</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Failed to dial: "</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">listener</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">Listen</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="s">"localhost:"</span><span class="o">+*</span><span class="nx">localPort</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">listener</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Like ssh -L by default, local connections are handled one at a time.</span><span class="w"></span>
<span class="w"> </span><span class="c1">// While one local connection is active in runTunnel, others will be stuck</span><span class="w"></span>
<span class="w"> </span><span class="c1">// dialing, waiting for this Accept.</span><span class="w"></span>
<span class="w"> </span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">listener</span><span class="p">.</span><span class="nx">Accept</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Issue a dial to the remote server on our SSH client; here "localhost"</span><span class="w"></span>
<span class="w"> </span><span class="c1">// refers to the remote server.</span><span class="w"></span>
<span class="w"> </span><span class="nx">remote</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="s">"localhost:"</span><span class="o">+*</span><span class="nx">remotePort</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"tunnel established with"</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="p">.</span><span class="nx">LocalAddr</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="nx">runTunnel</span><span class="p">(</span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">remote</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p><tt class="docutils literal">createSshConfig</tt> is the same as before. This program creates an SSH client
and then listens on a local socket on the provided port. For each connection,
it dials the "remote" port on through the SSH client and establishes a tunnel
between the two connections. <tt class="docutils literal">runTunnel</tt> is implemented like this:</p>
<div class="highlight"><pre><span></span><span class="c1">// runTunnel runs a tunnel between two connections; as soon as one connection</span><span class="w"></span>
<span class="c1">// reaches EOF or reports an error, both connections are closed and this</span><span class="w"></span>
<span class="c1">// function returns.</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="nx">runTunnel</span><span class="p">(</span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">remote</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">Conn</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">local</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">remote</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">done</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">make</span><span class="p">(</span><span class="kd">chan</span><span class="w"> </span><span class="kd">struct</span><span class="p">{},</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Copy</span><span class="p">(</span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">remote</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">done</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}{}</span><span class="w"></span>
<span class="w"> </span><span class="p">}()</span><span class="w"></span>
<span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Copy</span><span class="p">(</span><span class="nx">remote</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">done</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kd">struct</span><span class="p">{}{}</span><span class="w"></span>
<span class="w"> </span><span class="p">}()</span><span class="w"></span>
<span class="w"> </span><span class="o"><-</span><span class="nx">done</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>It uses goroutines to copy data in both directions between two connections, and
closes both connections as soon as one of them reaches EOF or reports an error.
Instead of running <tt class="docutils literal">ssh <span class="pre">-N</span> <span class="pre">-L</span></tt> to establish the tunnel, we can now run this
Go program with the same effect:</p>
<div class="highlight"><pre><span></span>$ go run ssh-local-tunnel.go -addr 159.89.238.232:22 -user root \
-keyfile ~/.ssh/id_ed25519 \
-rport 7780 \
-lport 7777
</pre></div>
<p>The <tt class="docutils literal">ssh</tt> package is a really nice showcase of the power of Go interfaces;
note how natural it appears in user code: just like a regular <tt class="docutils literal">net.Dial</tt> to a
TCP address, <tt class="docutils literal">ssh.Client.Dial</tt> returns a value implementing the <tt class="docutils literal">net.Conn</tt>
interface. For the user, it's completely seamless - one <tt class="docutils literal">net.Conn</tt> is as good
as another, even if they're quite different underneath (one is a direct wrapper
around a socket, the other a logical protocol layered on top of SSH).</p>
</div>
<div class="section" id="remote-port-forwarding">
<h2>Remote port forwarding</h2>
<p>We've discussed how local port forwarding works, what it can be used for and
how to set it up. Now let's talk about its complement - <em>remote</em> port
forwarding.</p>
<p>Suppose you have a web application running locally on your machine and
you want to test it from the public internet. Sure, you can access it from your
browser at <tt class="docutils literal">localhost</tt> - but that's just a simple scenario. Suppose it's a
backend for another online service and you really need a public address for it.
But your local machine is behind a NAT, so that's pretty hard. At this point you
will probably reach for a tool like <a class="reference external" href="https://ngrok.com/">ngrok</a> that creates
a tunnel between your local application and some public-accessible path. Well,
it turns out we don't actually need special tools here - <tt class="docutils literal">ssh</tt> can do the
job <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>.</p>
<p>This is what a "remote tunnel" does - similarly to the local tunnel it creates
a connection between a local port and a remote port, but the roles are flipped.
<tt class="docutils literal">sshd</tt> will listen to connections on the <em>remote</em> port and route them to
the <em>local</em> one. Here's a diagram:</p>
<img alt="Remote port forwarding flow of events" class="align-center" src="https://eli.thegreenplace.net/images/2022/remote-port-forwarding.png" />
<p>We have a program listening on port N locally, and we create a remote tunnel
using the <tt class="docutils literal">ssh</tt> client. From this point on, <tt class="docutils literal">sshd</tt> on the remote machine
starts listening to connections on port M.</p>
<p>Whenever a connection to port M on the remote machine is made, <tt class="docutils literal">sshd</tt> forwards
it to our local machine across the established tunnel, and our <tt class="docutils literal">ssh</tt> client,
in turn, forwards it to the local service on port N.</p>
<p>Let's see how to set it up using standard tools before we jump to the Go
implementation. This requires some preparation, because of obvious security
reasons. We have to tell our remote machine that it's OK for external clients
to access one of its ports, and we have to tell <tt class="docutils literal">sshd</tt> on that machine that
it's OK to forward ports.</p>
<p>For demonstration I'll use port 23000 on the remote machine; if you're using
<tt class="docutils literal">ufw</tt>, follow these instructions (if you have a different firewall YMMV, or
if you don't have a firewall at all feel free to skip):</p>
<div class="highlight"><pre><span></span># ufw allow 23000
Rule added
Rule added (v6)
# ufw status
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
23000 ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
23000 (v6) ALLOW Anywhere (v6)
# ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
</pre></div>
<p>At this point it may be worth testing that it actually works by running an
<tt class="docutils literal">nc</tt> server listening on port 23000 and accessing it from your local machine.</p>
<p>The second thing we should do is tell <tt class="docutils literal">sshd</tt> it's OK to forward ports. The
setting is called <tt class="docutils literal">GatewayPorts</tt> and it should be set to <tt class="docutils literal">yes</tt> in our
<tt class="docutils literal">/etc/ssh/sshd_config</tt> file on the remote machine. See <tt class="docutils literal">man sshd_config</tt>
for more details.</p>
<p>Assuming this was done, we're now ready to forward ports. Locally, I'll run
a simple <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/http-server-debug-request-headers.go">debugging HTTP server</a>
listening on port 8080:</p>
<div class="highlight"><pre><span></span>$ go run http-server-debug-request-headers.go
2022/11/16 22:32:54 Starting server on 127.0.0.1:8080
</pre></div>
<p>And in a separate terminal the <tt class="docutils literal">ssh</tt> command that sets up remote port
forwarding (<tt class="docutils literal"><span class="pre">-R</span></tt> option) from remote port 23000 to local port 8080:</p>
<div class="highlight"><pre><span></span>$ ssh -N -R 23000:localhost:8080 root@159.89.238.232
</pre></div>
<p>Now we're all set! A <tt class="docutils literal">curl</tt> from the local machine should now hit my debugging
server through a public IP address:</p>
<div class="highlight"><pre><span></span>$ curl http://159.89.238.232:23000/hi/there
hello /hi/there
</pre></div>
<p>And we can easily access it from a browser as well. It's a bit funny that we're
running a local client to access a local server through a remote machine! Feel
free to prove to yourself that this is "real" by accessing the same address
from a different computer (like your phone!).</p>
<p>Now let's turn to the Go implementation. A lot of its code will be shared with
the previous samples, so I'll just show what's different - the <tt class="docutils literal">main</tt> function
itself:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"addr"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"ssh server address to dial as <hostname>:<port>"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">username</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"user"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"username for ssh"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">keyFile</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"keyfile"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"file with private key for SSH authentication"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">remotePort</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"rport"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"remote port for tunnel"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">localPort</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"lport"</span><span class="p">,</span><span class="w"> </span><span class="s">""</span><span class="p">,</span><span class="w"> </span><span class="s">"local port for tunnel"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">config</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">createSshConfig</span><span class="p">(</span><span class="o">*</span><span class="nx">username</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">keyFile</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">client</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">ssh</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">config</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"Failed to dial: "</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">listener</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">client</span><span class="p">.</span><span class="nx">Listen</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="s">"localhost:"</span><span class="o">+*</span><span class="nx">remotePort</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">listener</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">remote</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">listener</span><span class="p">.</span><span class="nx">Accept</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="kd">func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="s">"localhost:"</span><span class="o">+*</span><span class="nx">localPort</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"tunnel established with"</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="p">.</span><span class="nx">LocalAddr</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="nx">runTunnel</span><span class="p">(</span><span class="nx">local</span><span class="p">,</span><span class="w"> </span><span class="nx">remote</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}()</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>In spirit, the main server loop is similar to the local tunnel example, but
there are a couple of key differences:</p>
<ol class="arabic simple">
<li>Whereas the local tunnel listens on the local port and creates remote
connections in response to local connections, in the remote case these
roles are inverted: we listen for remote connections, and spawn local
connections in response.</li>
<li>This tunnel implementation supports more concurrency since we want to be
able to handle multiple remote clients connecting to our local server
simultaneously. Therefore, a new goroutine is spun up in response to new
remote connections.</li>
</ol>
</div>
<div class="section" id="appendix-how-ssh-implements-tunnels">
<h2>Appendix: how SSH implements tunnels</h2>
<p>The standard networking stack is all about layering and multiplexing. When we
create sockets between services it feels very convenient and natural, but lower
level protocols (like IP) have no notion of connections or even ports. They just
send packets that could get lost, corrupted or arrive out of order. Transport
protocols like TCP are layered on top of these packets and provide additional
abstractions.</p>
<p>Two computers connected together may have the illusion of communicating over
several sockets (and ports) simultaneously, while in reality they're just
sending IP packets there and back.</p>
<p>SSH is no different. It's an application layer protocol sitting on top of TCP.
Deep down, there's a single TCP socket (usually on port 22) for exchanging
information. Once the cryptographic setup is finished and the channel is secure,
SSH gives users the illusion of multiple communications happening
simultaneously, but it's just packets flowing over the socket. You can have
multiple interactive SSH terminals running at the same time, and multiple
"channels" - which is SSH's term for data streams that are used for port
forwarding.</p>
<p>The SSH protocol itself is described in <a class="reference external" href="https://www.rfc-editor.org/rfc/rfc4251">RFC 4251</a>, and its <em>connection protocol</em> is
described in <a class="reference external" href="https://www.rfc-editor.org/rfc/rfc4254">RFC 4254</a>. This is the
part of the protocol that takes care of multiplexing multiple communication
streams on top of the single encrypted socket SSH establishes. Here's a quote
from RFC 4254:</p>
<blockquote>
<p>All terminal sessions, forwarded connections, etc., are channels.
Either side may open a channel. Multiple channels are multiplexed
into a single connection.</p>
<p>Channels are identified by numbers at each end. The number referring
to a channel may be different on each side. Requests to open a
channel contain the sender's channel number. Any other channel-
related messages contain the recipient's channel number for the
channel.</p>
<p>Channels are flow-controlled. No data may be sent to a channel until
a message is received to indicate that window space is available.</p>
</blockquote>
<p>Forwarded ports (tunnels) are mapped directly onto these SSH channels. Each
forwarded port gets a channel, and data sent to these ports is encapsulated in
SSH connection protocol packets with the appropriate channel number. The SSH
client or server on the other end unpacks this packet, looks at the channel
number and sends the data down the port corresponding to the channel.</p>
<p>There are many additional sources of information on how SSH channels work. I
found these two particularly useful:</p>
<ul class="simple">
<li><a class="reference external" href="https://dev.to/progrium/the-history-and-future-of-socket-level-multiplexing-1d5n">This post</a>
discusses more of the philosophy behind how this works and provides
interesting context. If you're interested in port forwarding, I strongly
recommend reading this one.</li>
<li><a class="reference external" href="https://ophirharpaz.com/posts/how-does-ssh-port-forwarding-work/">This one</a> dives deep into the code
of a specific SSH implementation (DropBear), with C code snippets showing how
channels are implemented.</li>
</ul>
<p>Finally, I want to note that the idea of protocol multiplexing keeps being
reused in networking protocols. HTTP/2 uses multiplexing (with similar
packet-level encapsulation on top of TLS/TCP) to mix simultaneous multiple
connections and features like server push. QUIC (HTTP/3) takes it one step
farther and multiplexes everything on top of UDP, doing away with TCP
connections altogether.</p>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>As before, this presupposes that the local machine possesses a private
SSH key for the <tt class="docutils literal">root</tt> user on the VPS; the public dual of this key
is supposed to be in <tt class="docutils literal">.ssh/authorized_keys</tt> on the remote machine.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td><p class="first">Interestingly, the original implementation of ngrok was based upon a
tool called <tt class="docutils literal">localtunnel</tt> which itself was just a script wrapping
<tt class="docutils literal">ssh</tt>, using an approach that's very similar to the one
presented in this post.</p>
<p class="last">This is not to say that ngrok isn't useful! It has a large number of
very convenient features that go above and beyond the simple tunnels I'm
showing here.</p>
</td></tr>
</tbody>
</table>
</div>
Go and Proxy Servers: Part 2 - HTTPS Proxies2022-11-09T19:31:00-08:002022-12-15T13:27:28-08:00Eli Benderskytag:eli.thegreenplace.net,2022-11-09:/2022/go-and-proxy-servers-part-2-https-proxies/<p>This is the second post in a series about proxy servers and Go. Here is a list
of posts in the series:</p>
<ul class="simple">
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">Part 1 - HTTP Proxies</a></li>
<li>Part 2 - HTTPS Proxies (this part)</li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-3-socks-proxies/">Part 3 - SOCKS Proxies</a></li>
</ul>
<p>The previous part is an overview of proxy servers and presents some basic Go …</p><p>This is the second post in a series about proxy servers and Go. Here is a list
of posts in the series:</p>
<ul class="simple">
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">Part 1 - HTTP Proxies</a></li>
<li>Part 2 - HTTPS Proxies (this part)</li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-3-socks-proxies/">Part 3 - SOCKS Proxies</a></li>
</ul>
<p>The previous part is an overview of proxy servers and presents some basic Go
implementations, as well as discussing how to configure Go clients to use
proxies. While it serves as important background information, it has one
glaring omission: these days most web traffic is over HTTPS, not HTTP.</p>
<p>This post will cover proxying HTTPS. It assumes you know what HTTPS and TLS are
(if not, check out this earlier <a class="reference external" href="https://eli.thegreenplace.net/2021/go-https-servers-with-tls/">post on HTTP servers in Go</a>).</p>
<div class="section" id="what-is-different">
<h2>What is different?</h2>
<p>Why won't HTTP proxies "just work" for HTTPS? The reason is that an HTTPS client
expects to talk to a specific server, and will look for a valid certificate from
that server to start sending information.</p>
<p>Say we want to access <a class="reference external" href="https://example.org">https://example.org</a>; when our client initiates an HTTPS
session to this domain, it expects a valid signed certificate for
<a class="reference external" href="https://example.org">https://example.org</a>.
A proxy server unaffiliated with this domain will find it difficult to provide
such a certificate.</p>
<p>Proxies work for HTTPS (HTTP over TLS) by doing one of the following (or a
variation):</p>
<p><strong>First</strong>, the proxy can "terminate" the TLS connection <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>; the proxy here
<em>is</em> the server. It's deployed by the developers of the domain we're accessing,
and thus has the right certificates. This is commonly done for
reverse proxies. For our <a class="reference external" href="https://example.org">https://example.org</a> example - the proxy would have a
valid certificate for this domain and would be able to talk to clients. What the
proxy does on the other side is up to it - it could be using unencrypted HTTP
with backend servers, or use HTTPS with some internal certificates the external
world doesn't need to know about.</p>
<p>Here's a Wikipedia diagram of this setup:</p>
<img alt="Diagram showing a TLS termination proxy, taken from Wikipedia" class="align-center" src="https://eli.thegreenplace.net/images/2022/tls-termination-proxy.png" style="width: 500px;" />
<p><strong>Second</strong>, the proxy can tunnel the TLS connection to the target server. In
this scenario the proxy acts as a blind pipe for traffic.</p>
<p><strong>Third</strong>, the proxy can tunnel the TLS connection to the target while also
reading (and potentially modifying) its contents. It's still a tunnel, but not
a blind one. If you wonder how this is even possible, read on.</p>
</div>
<div class="section" id="tls-terminating-reverse-proxy">
<h2>TLS-terminating reverse proxy</h2>
<p>Let's start with the simplest case - a TLS-terminating reverse proxy. This is
very similar to the basic HTTP reverse proxy example from the
<a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">previous post</a>,
but it listens to HTTPS traffic on its incoming port. This server will need a
certificate for localhost that the client trusts; I recommend <a class="reference external" href="https://github.com/FiloSottile/mkcert">mkcert</a>, or see <a class="reference external" href="https://eli.thegreenplace.net/2021/go-https-servers-with-tls/">this post</a> for more
details. Here's the code (full code sample <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/https-reverse-proxy.go">on GitHub</a>):</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">fromAddr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"from"</span><span class="p">,</span><span class="w"> </span><span class="s">"127.0.0.1:9090"</span><span class="p">,</span><span class="w"> </span><span class="s">"proxy's listening address"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">toAddr</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"to"</span><span class="p">,</span><span class="w"> </span><span class="s">"127.0.0.1:8080"</span><span class="p">,</span><span class="w"> </span><span class="s">"the address this proxy will forward to"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">certFile</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"certfile"</span><span class="p">,</span><span class="w"> </span><span class="s">"cert.pem"</span><span class="p">,</span><span class="w"> </span><span class="s">"certificate PEM file"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">keyFile</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"keyfile"</span><span class="p">,</span><span class="w"> </span><span class="s">"key.pem"</span><span class="p">,</span><span class="w"> </span><span class="s">"key PEM file"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">toUrl</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">parseToUrl</span><span class="p">(</span><span class="o">*</span><span class="nx">toAddr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">proxy</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httputil</span><span class="p">.</span><span class="nx">NewSingleHostReverseProxy</span><span class="p">(</span><span class="nx">toUrl</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">srv</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">http</span><span class="p">.</span><span class="nx">Server</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">Addr</span><span class="p">:</span><span class="w"> </span><span class="o">*</span><span class="nx">fromAddr</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Handler</span><span class="p">:</span><span class="w"> </span><span class="nx">proxy</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">TLSConfig</span><span class="p">:</span><span class="w"> </span><span class="o">&</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">MinVersion</span><span class="p">:</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nx">VersionTLS13</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">PreferServerCipherSuites</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Starting proxy server on"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">fromAddr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">srv</span><span class="p">.</span><span class="nx">ListenAndServeTLS</span><span class="p">(</span><span class="o">*</span><span class="nx">certFile</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">keyFile</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"ListenAndServe:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>To see this proxy in action, we'll need a bit of setup. First, as discussed
<a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">previously</a>,
I have <tt class="docutils literal">local.alias</tt> set up as an alias for 127.0.0.1 <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>; I then used
<tt class="docutils literal">mkcert</tt> to generate a locally-trusted certificate with an accompanying
private key. Then, the proxy can be run:</p>
<div class="highlight"><pre><span></span>$ go run https-reverse-proxy.go \
-certfile <path to certificate .pem> \
-keyfile <path to key .pem>
2022/11/04 19:43:56 Starting proxy server on 127.0.0.1:9090
</pre></div>
<p>In a separate window, run <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/http-server-debug-request-headers.go">the debugging HTTP server</a>
listening on its default port 8080:</p>
<div class="highlight"><pre><span></span>$ go run http-server-debug-request-headers.go
2022/11/04 19:44:18 Starting server on 127.0.0.1:8080
</pre></div>
<p>Finally, we can issue <tt class="docutils literal">curl</tt> requests to our proxy server, and they will be
properly forwarded to the debugging server:</p>
<div class="highlight"><pre><span></span>$ curl https://local.alias:9090/fo/540
hello /fo/540
</pre></div>
<p>As mentioned earlier, this proxy is called "TLS terminating" because it takes
care of the encrypted TLS communication with clients. In a realistic scenario,
this server would be serving <a class="reference external" href="https://mybusiness.com">https://mybusiness.com</a> at the default HTTPS port
443, and would have the valid certificate for this domain. This is how most
non-trivial websites run these days, and lots of production-grade tools exist
to support such a workflow. You can easily configure web servers like Caddy and
Nginx to serve as TLS-terminating reverse proxies.</p>
</div>
<div class="section" id="a-proxy-for-tunneling-arbitrary-traffic-to-destination">
<h2>A proxy for tunneling arbitrary traffic to destination</h2>
<p>As we've seen, HTTPS doesn't present a big issue for reverse proxies, because
these just happen to be the right place to terminate the TLS connection anyhow.
How about forward proxies, though? These seem to have a real problem with HTTPS
because they don't have the right certificates.</p>
<p>The solution IETF came up with is a special HTTP method called <tt class="docutils literal">CONNECT</tt>.
Quoting from <a class="reference external" href="https://www.rfc-editor.org/rfc/rfc7231#section-4.3.6">RFC 7231</a>:</p>
<!-- -->
<blockquote>
<p>The CONNECT method requests that the recipient establish a tunnel to
the destination origin server identified by the request-target and,
if successful, thereafter restrict its behavior to blind forwarding
of packets, in both directions, until the tunnel is closed. Tunnels
are commonly used to create an end-to-end virtual connection, through
one or more proxies, which can then be secured using TLS (Transport
Layer Security, [RFC5246]).</p>
<p>CONNECT is intended only for use in requests to a proxy. An origin
server that receives a CONNECT request for itself MAY respond with a
2xx (Successful) status code to indicate that a connection is
established. However, most origin servers do not implement CONNECT.</p>
</blockquote>
<p>To create a tunnel, the following sequence of events occurs:</p>
<ol class="arabic simple">
<li>The client contacts the proxy and sends it a <tt class="docutils literal">CONNECT</tt> request. The
request specifies which destination server (host and port) to connect to, for
example <tt class="docutils literal">CONNECT example.org:443 HTTP/1.1</tt></li>
<li>The proxy establishes a TCP connection to the specified host:port</li>
<li>If successful, it returns a <tt class="docutils literal">HTTP/1.1 200 OK</tt> response to the client</li>
<li>From this point on, the proxy will forward all TCP traffic between the client
and destination server on this connection. This traffic is often HTTPS, but
it could be anything else</li>
<li>The proxy monitors both sides of the connection and terminates the tunnel
as soon as one of the sides closes its connection.</li>
</ol>
<img alt="Diagram showing a CONNECT tunneling proxy" class="align-center" src="https://eli.thegreenplace.net/images/2022/connect-tunnel-proxy.png" />
<p>As you'd expect by now, implementing such a proxy in Go is fairly
straightforward! <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/connect-tunnel-proxy.go">The full code sample is on GitHub</a>;
let's start with the standard scaffolding <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a>:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">addr</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">String</span><span class="p">(</span><span class="s">"addr"</span><span class="p">,</span><span class="w"> </span><span class="s">"127.0.0.1:9999"</span><span class="p">,</span><span class="w"> </span><span class="s">"proxy address"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">flag</span><span class="p">.</span><span class="nx">Parse</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">proxy</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">forwardProxy</span><span class="p">{}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"Starting proxy server on"</span><span class="p">,</span><span class="w"> </span><span class="o">*</span><span class="nx">addr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ListenAndServe</span><span class="p">(</span><span class="o">*</span><span class="nx">addr</span><span class="p">,</span><span class="w"> </span><span class="nx">proxy</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"ListenAndServe:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">type</span><span class="w"> </span><span class="nx">forwardProxy</span><span class="w"> </span><span class="kd">struct</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>To have <tt class="docutils literal">forwardProxy</tt> implement the <tt class="docutils literal">http.Handler</tt> interface, let's add
a <tt class="docutils literal">ServeHTTP</tt> method:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="o">*</span><span class="nx">forwardProxy</span><span class="p">)</span><span class="w"> </span><span class="nx">ServeHTTP</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Method</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">MethodConnect</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">proxyConnect</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="s">"this proxy only supports CONNECT"</span><span class="p">,</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusMethodNotAllowed</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This proxy will only agree to serve <tt class="docutils literal">CONNECT</tt> tunnels and won't work as a
regular HTTP proxy. Implementing the latter is left as an exercise to the
reader. The key function is <tt class="docutils literal">proxyConnect</tt>:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">proxyConnect</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"CONNECT requested to %v (from %v)"</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Host</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">RemoteAddr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">targetConn</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Host</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"failed to dial to target"</span><span class="p">,</span><span class="w"> </span><span class="nx">req</span><span class="p">.</span><span class="nx">Host</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">Error</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">.</span><span class="nx">Error</span><span class="p">(),</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusServiceUnavailable</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">w</span><span class="p">.</span><span class="nx">WriteHeader</span><span class="p">(</span><span class="nx">http</span><span class="p">.</span><span class="nx">StatusOK</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">hj</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">w</span><span class="p">.(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Hijacker</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"http server doesn't support hijacking connection"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">clientConn</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">hj</span><span class="p">.</span><span class="nx">Hijack</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"http hijacking failed"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"tunnel established"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="nx">tunnelConn</span><span class="p">(</span><span class="nx">targetConn</span><span class="p">,</span><span class="w"> </span><span class="nx">clientConn</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">go</span><span class="w"> </span><span class="nx">tunnelConn</span><span class="p">(</span><span class="nx">clientConn</span><span class="p">,</span><span class="w"> </span><span class="nx">targetConn</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This code reads the destination address from the request and establishes a new
TCP connection. It then... <em>hijacks</em> the client connection? What's that all
about? Don't worry, it's simpler than it sounds! When working with the
<tt class="docutils literal">net/http</tt> package, we don't deal with TCP connections directly; instead, we
deal with abstractions like <tt class="docutils literal">ResponseWriter</tt>. But underlying every HTTP
session is a TCP connection; the <tt class="docutils literal">Hijack</tt> method lets us get to that
connection, essentially puncturing a hole through the abstraction.</p>
<p>The caveat is that once we've hijacked the connection, we're on our own. The
<tt class="docutils literal">net/http</tt> package will no longer manage things for us; we have to close the
connection on our own when we're done, and so on.</p>
<p>In this case of tunneling traffic in the proxy, the raw TCP connection is
exactly what we need. So we end up with two TCP connections - one with the
client and one with the destination server. The next step is to hook them up
together. We start two goroutines - one for each direction; <tt class="docutils literal">tunnelConn</tt>
does this:</p>
<div class="highlight"><pre><span></span><span class="kd">func</span><span class="w"> </span><span class="nx">tunnelConn</span><span class="p">(</span><span class="nx">dst</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">WriteCloser</span><span class="p">,</span><span class="w"> </span><span class="nx">src</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">ReadCloser</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">Copy</span><span class="p">(</span><span class="nx">dst</span><span class="p">,</span><span class="w"> </span><span class="nx">src</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">dst</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="nx">src</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>For a production server you'd probably want to be a bit more careful w.r.t
error handling, but this will do for demonstration purposes.</p>
<p>Let's take this proxy for a spin; first, run it in a terminal:</p>
<div class="highlight"><pre><span></span>$ go run connect-tunnel-proxy.go
2022/11/04 21:10:49 Starting proxy server on 127.0.0.1:9999
</pre></div>
<p>Now we can access an HTTPS site using this proxy; we'll invoke <tt class="docutils literal">curl <span class="pre">-v</span></tt> to
see exactly what it's doing. Note that we set up the <tt class="docutils literal">https_proxy</tt> env var
to tell <tt class="docutils literal">curl</tt> which proxy it needs to contact in order to access HTTPS
targets:</p>
<div class="highlight"><pre><span></span>$ https_proxy=localhost:9999 curl -v https://example.org
* Uses proxy env variable https_proxy == 'localhost:9999'
* Trying 127.0.0.1:9999...
* Connected to (nil) (127.0.0.1) port 9999 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to example.org:443
> CONNECT example.org:443 HTTP/1.1
> Host: example.org:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
* Ignoring Transfer-Encoding in CONNECT 200 response
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
...
... // more TLS spew
...
<!doctype html>
... // And the HTML for https://example.org is dumped
</pre></div>
<p>We see that <tt class="docutils literal">curl</tt> contacted our proxy, sending it a <tt class="docutils literal">CONNECT</tt> request for
<tt class="docutils literal"><span class="pre">https://example.org</span></tt>. Our proxy replied with <tt class="docutils literal">200 OK</tt> and then <tt class="docutils literal">curl</tt>
proceeded to perform a TLS handshake with the destination server, through the
proxy. It works! Our proxy also prints some useful log lines:</p>
<div class="highlight"><pre><span></span>2022/11/04 21:12:15 CONNECT requested to example.org:443 (from 127.0.0.1:33468)
2022/11/04 21:12:15 tunnel established
</pre></div>
<p>In <a class="reference external" href="https://eli.thegreenplace.net/2022/go-and-proxy-servers-part-1-http-proxies/">Part 1</a>,
we've written a simple HTTP client in Go to test our proxy interactions. Let's
reuse it here, this time providing a <tt class="docutils literal">https_proxy</tt> setting:</p>
<div class="highlight"><pre><span></span>$ https_proxy=localhost:9999 go run http-get-basic.go --target https://example.org
Response status: 200 OK
<!doctype html>
... // And the HTML for https://example.org is dumped
</pre></div>
</div>
<div class="section" id="mitm-intercepting-https-proxies">
<h2>MITM / intercepting HTTPS proxies</h2>
<p>So far in our discussion of HTTPS proxies we've assumed that proxies cannot
actually snoop into the underlying HTTPS traffic unless they have the right
certificate. This is true, of course. I'm not aware of any breach in the
security of a modern version of TLS (1.3 at the time of writing). However, in
cryptography what matters is the weakest link - and the disclaimer "unless they
have the right certificate" hints at one.</p>
<p><a class="reference external" href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">Man-in-the-middle (MITM)</a> forward proxies
allow intercepting HTTPS traffic, reading and even modifying it. They accomplish
this by enjoying the collaboration of the system administrator of the machine
the client is running on.</p>
<p>TLS certificates follow a "chain of trust". With access to the client machine,
we can install a special root certificate authority (CA) that the machine
inherently trusts, and use it to sign fake certificates the proxy can generate
for any website.</p>
<p>Here's how it works:</p>
<ul class="simple">
<li>The client machine trusts certificate authority X (this was set up by the
administrator or by the user themselves). Typically browsers come with
hard-coded lists of "root CAs", but it's possible to augment these lists
with system-specific settings</li>
<li>A tunneling proxy has the private keys of X, and can use them to sign
certificates</li>
<li>Once the proxy receives a <tt class="docutils literal">CONNECT</tt> request to access some <tt class="docutils literal">domain.com</tt>,
it generates a fake certificate for this domain, signing it with X's key</li>
<li>This permits the proxy to communicate with the client on behalf of
<tt class="docutils literal">domain.com</tt>, essentially decrypting all traffic intended only for that
domain's eyes</li>
<li>The proxy can look at the traffic and modify it, and then forward it to the
actual <tt class="docutils literal">domain.com</tt> (using a separate TLS connection), and similarly examine
and modify the responses</li>
</ul>
<p>This sounds sinister, but that isn't the only use case. Debugging proxies exist
that let us do this for valid reasons. Moreover, if an organization
installs a proxy it probably doesn't want users to easily circumvent it by
blind tunneling, so it may set all machines up such that the proxy has access
to the traffic's contents. If you think your bank logins are safe from your
work laptop because the bank's website uses TLS, think again. It's unlikely that
your workplace will use your bank credentials for any sinister purposes, but
they probably could if they wanted to.</p>
<p>Note that in some setups, the organization doesn't even have to set up a proxy
to do this. If all the traffic from a computer goes through a specific set of
routers, for example, these routers can implement the same trick to decrypt
and proxy all HTTPS traffic - without the user suspecting anything. The critical
part here is having the private keys for a certificate authority the user's
machine trusts.</p>
<p>Implementing such a proxy in Go is not hard, but there are some nuances to be
aware of. I have a <a class="reference external" href="https://github.com/eliben/code-for-blog/blob/master/2022/go-and-proxies/connect-mitm-proxy.go">working implementation on GitHub</a>.
Here's the well-commented main HTTP handler of the proxy (for the functions it
calls,
consult the full source):</p>
<div class="highlight"><pre><span></span><span class="c1">// proxyConnect implements the MITM proxy for CONNECT tunnels.</span><span class="w"></span>
<span class="kd">func</span><span class="w"> </span><span class="p">(</span><span class="nx">p</span><span class="w"> </span><span class="o">*</span><span class="nx">mitmProxy</span><span class="p">)</span><span class="w"> </span><span class="nx">proxyConnect</span><span class="p">(</span><span class="nx">w</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ResponseWriter</span><span class="p">,</span><span class="w"> </span><span class="nx">proxyReq</span><span class="w"> </span><span class="o">*</span><span class="nx">http</span><span class="p">.</span><span class="nx">Request</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"CONNECT requested to %v (from %v)"</span><span class="p">,</span><span class="w"> </span><span class="nx">proxyReq</span><span class="p">.</span><span class="nx">Host</span><span class="p">,</span><span class="w"> </span><span class="nx">proxyReq</span><span class="p">.</span><span class="nx">RemoteAddr</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// "Hijack" the client connection to get a TCP (or TLS) socket we can read</span><span class="w"></span>
<span class="w"> </span><span class="c1">// and write arbitrary data to/from.</span><span class="w"></span>
<span class="w"> </span><span class="nx">hj</span><span class="p">,</span><span class="w"> </span><span class="nx">ok</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">w</span><span class="p">.(</span><span class="nx">http</span><span class="p">.</span><span class="nx">Hijacker</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">!</span><span class="nx">ok</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"http server doesn't support hijacking connection"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">clientConn</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">hj</span><span class="p">.</span><span class="nx">Hijack</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"http hijacking failed"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// proxyReq.Host will hold the CONNECT target host, which will typically have</span><span class="w"></span>
<span class="w"> </span><span class="c1">// a port - e.g. example.org:443</span><span class="w"></span>
<span class="w"> </span><span class="c1">// To generate a fake certificate for example.org, we have to first split off</span><span class="w"></span>
<span class="w"> </span><span class="c1">// the host from the port.</span><span class="w"></span>
<span class="w"> </span><span class="nx">host</span><span class="p">,</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">net</span><span class="p">.</span><span class="nx">SplitHostPort</span><span class="p">(</span><span class="nx">proxyReq</span><span class="p">.</span><span class="nx">Host</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"error splitting host/port:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Create a fake TLS certificate for the target host, signed by our CA. The</span><span class="w"></span>
<span class="w"> </span><span class="c1">// certificate will be valid for 10 days - this number can be changed.</span><span class="w"></span>
<span class="w"> </span><span class="nx">pemCert</span><span class="p">,</span><span class="w"> </span><span class="nx">pemKey</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">createCert</span><span class="p">([]</span><span class="kt">string</span><span class="p">{</span><span class="nx">host</span><span class="p">},</span><span class="w"> </span><span class="nx">p</span><span class="p">.</span><span class="nx">caCert</span><span class="p">,</span><span class="w"> </span><span class="nx">p</span><span class="p">.</span><span class="nx">caKey</span><span class="p">,</span><span class="w"> </span><span class="mi">240</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="nx">tlsCert</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nx">X509KeyPair</span><span class="p">(</span><span class="nx">pemCert</span><span class="p">,</span><span class="w"> </span><span class="nx">pemKey</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Send an HTTP OK response back to the client; this initiates the CONNECT</span><span class="w"></span>
<span class="w"> </span><span class="c1">// tunnel. From this point on the client will assume it's connected directly</span><span class="w"></span>
<span class="w"> </span><span class="c1">// to the target.</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">_</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">clientConn</span><span class="p">.</span><span class="nx">Write</span><span class="p">([]</span><span class="nb">byte</span><span class="p">(</span><span class="s">"HTTP/1.1 200 OK\r\n\r\n"</span><span class="p">));</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"error writing status to client:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Configure a new TLS server, pointing it at the client connection, using</span><span class="w"></span>
<span class="w"> </span><span class="c1">// our certificate. This server will now pretend being the target.</span><span class="w"></span>
<span class="w"> </span><span class="nx">tlsConfig</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="o">&</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Config</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">PreferServerCipherSuites</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">CurvePreferences</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nx">tls</span><span class="p">.</span><span class="nx">CurveID</span><span class="p">{</span><span class="nx">tls</span><span class="p">.</span><span class="nx">X25519</span><span class="p">,</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nx">CurveP256</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nx">MinVersion</span><span class="p">:</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nx">VersionTLS13</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nx">Certificates</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="nx">tls</span><span class="p">.</span><span class="nx">Certificate</span><span class="p">{</span><span class="nx">tlsCert</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nx">tlsConn</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">tls</span><span class="p">.</span><span class="nx">Server</span><span class="p">(</span><span class="nx">clientConn</span><span class="p">,</span><span class="w"> </span><span class="nx">tlsConfig</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">tlsConn</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Create a buffered reader for the client connection; this is required to</span><span class="w"></span>
<span class="w"> </span><span class="c1">// use http package functions with this connection.</span><span class="w"></span>
<span class="w"> </span><span class="nx">connReader</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">bufio</span><span class="p">.</span><span class="nx">NewReader</span><span class="p">(</span><span class="nx">tlsConn</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Run the proxy in a loop until the client closes the connection.</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Read an HTTP request from the client; the request is sent over TLS that</span><span class="w"></span>
<span class="w"> </span><span class="c1">// connReader is configured to serve. The read will run a TLS handshake in</span><span class="w"></span>
<span class="w"> </span><span class="c1">// the first invocation (we could also call tlsConn.Handshake explicitly</span><span class="w"></span>
<span class="w"> </span><span class="c1">// before the loop, but this isn't necessary).</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Note that while the client believes it's talking across an encrypted</span><span class="w"></span>
<span class="w"> </span><span class="c1">// channel with the target, the proxy gets these requests in "plain text"</span><span class="w"></span>
<span class="w"> </span><span class="c1">// because of the MITM setup.</span><span class="w"></span>
<span class="w"> </span><span class="nx">r</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">ReadRequest</span><span class="p">(</span><span class="nx">connReader</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="nx">io</span><span class="p">.</span><span class="nx">EOF</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// We can dump the request; log it, modify it...</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">b</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httputil</span><span class="p">.</span><span class="nx">DumpRequest</span><span class="p">(</span><span class="nx">r</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"incoming request:\n%s\n"</span><span class="p">,</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">b</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Take the original request and changes its destination to be forwarded</span><span class="w"></span>
<span class="w"> </span><span class="c1">// to the target server.</span><span class="w"></span>
<span class="w"> </span><span class="nx">changeRequestToTarget</span><span class="p">(</span><span class="nx">r</span><span class="p">,</span><span class="w"> </span><span class="nx">proxyReq</span><span class="p">.</span><span class="nx">Host</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Send the request to the target server and log the response.</span><span class="w"></span>
<span class="w"> </span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">http</span><span class="p">.</span><span class="nx">DefaultClient</span><span class="p">.</span><span class="nx">Do</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Fatal</span><span class="p">(</span><span class="s">"error sending request to target:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">b</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">httputil</span><span class="p">.</span><span class="nx">DumpResponse</span><span class="p">(</span><span class="nx">resp</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Printf</span><span class="p">(</span><span class="s">"target response:\n%s\n"</span><span class="p">,</span><span class="w"> </span><span class="nb">string</span><span class="p">(</span><span class="nx">b</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">defer</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">Body</span><span class="p">.</span><span class="nx">Close</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="c1">// Send the target server's response back to the client.</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">Write</span><span class="p">(</span><span class="nx">tlsConn</span><span class="p">);</span><span class="w"> </span><span class="nx">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="kc">nil</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nx">log</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"error writing response back:"</span><span class="p">,</span><span class="w"> </span><span class="nx">err</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>This proxy only handles the simple case where the target's domain is explicitly
provided in the <tt class="docutils literal">CONNECT</tt> request. It doesn't support x509 SAN extensions or
SNI, though these should be easy to add. For some additional details on the
various complications see <a class="reference external" href="https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/">this documentation page from mitmproxy</a>, a featureful
intercepting HTTPS proxy implementation in Python.</p>
<p>To run this proxy, we need to provide it with the path to the certificate and
private key of a CA implicitly trusted by our machine. Since I'm using
<tt class="docutils literal">mkcert</tt> and have previously ran it with <tt class="docutils literal"><span class="pre">-install</span></tt> to install such a CA, I
found the path to these by running <tt class="docutils literal">mkcert <span class="pre">-CAROOT</span></tt>. The invocation then looks
something like this (your paths will be different):</p>
<div class="highlight"><pre><span></span>$ mkcert -CAROOT
/home/eliben/.local/share/mkcert
$ go run connect-mitm-proxy.go \
-cacertfile /home/eliben/.local/share/mkcert/rootCA.pem \
-cakeyfile /home/eliben/.local/share/mkcert/rootCA-key.pem
2022/11/08 20:49:02 loaded CA certificate and key; IsCA=true
2022/11/08 20:49:02 Starting proxy server on 127.0.0.1:9999
</pre></div>
<hr class="docutils" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>The first time you hear it, the term "terminate" may sound a bit odd.
In this context it doesn't carry any negative connotations; it simply
means "the TLS ends here", to distinguish from "TLS pass-through",
where the TLS traffic is tunneled as-is to the backend server.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>You can get by with just <tt class="docutils literal">localhost</tt> too, though.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Note that while this proxy helps us access HTTPS destinations, it serves
over plain HTTP itself. Having this proxy serve HTTPS is straightforward,
and can be accomplished as an exercise (hint: see the earlier code
sample in this post).</td></tr>
</tbody>
</table>
</div>