Eli Bendersky's website - Internet

Reading Google Sheets from a Go program

2024-05-31T18:07:00-07:00

I recently needed to process some data from a Google Sheet in a Go program, and was looking for the most straightforward way to do so on my local machine. This post lists some approaches that I found to work, with full source code.

To access the Sheets API, you'll need a GCP project, and would typically have the gcloud command-line tool installed. To enable the sheets API for your project, run:

$ gcloud services enable sheets.googleapis.com --project=<PROJECT-NAME>

If you want to list which APIs are already enabled, you can do:

$ gcloud services list --enabled --project=<PROJECT-NAME>

The simplest approach I found to work was using a service account. This post demonstrates this approach, as well as a (slightly) more involved approach that uses Oauth 2.0

Service account

A service account on GCP can be thought of as a virtual account, along with its own email address, attached to a project. These accounts have their own auth, permissions, etc. This is very useful for running on a VM - you typically don't want the VM to be logged in with your primary Google account, and this service account can be specific to a given VM (or a group thereof).

Start by creating a new service account on this page. Once created, select Manage Keys in the Actions menu, and add a new key. This will download a private key to your machine; keep it safe! The following program expects this key file to be provided with the -keyfile flag:

package main

import (
  "context"
  "flag"
  "fmt"
  "io/ioutil"
  "log"

  "golang.org/x/oauth2/google"
  "google.golang.org/api/option"
  "google.golang.org/api/sheets/v4"
)

func main() {
  keyFilePath := flag.String("keyfile", "", "path to the credentials file")
  flag.Parse()

  ctx := context.Background()
  credentials, err := ioutil.ReadFile(*keyFilePath)
  if err != nil {
    log.Fatal("unable to read key file:", err)
  }

  scopes := []string{
    "https://www.googleapis.com/auth/spreadsheets.readonly",
  }
  config, err := google.JWTConfigFromJSON(credentials, scopes...)
  if err != nil {
    log.Fatal("unable to create JWT configuration:", err)
  }

  srv, err := sheets.NewService(ctx, option.WithHTTPClient(config.Client(ctx)))
  if err != nil {
    log.Fatalf("unable to retrieve sheets service: %v", err)
  }

  // ...

We can specify the requested scopes (permissions) when creating an auth config. Here we're asking for read-only access to the Google Sheets.

Once auth succeeds (sheets.NewService returns w/o an error), we can use the sheets package to read and analyze the sheet; the code below simply prints the document's title and emits all the values from columns A and B in Sheet1.

  docId := "1qsNWsZuw98r9HEl01vwxCO5O1sIsI-fr0bJ4KGVvWsU"
  doc, err := srv.Spreadsheets.Get(docId).Do()
  if err != nil {
    log.Fatalf("unable to retrieve data from document: %v", err)
  }
  fmt.Printf("The title of the doc is: %s\n", doc.Properties.Title)

  val, err := srv.Spreadsheets.Values.Get(docId, "Sheet1!A:B").Do()
  if err != nil {
    log.Fatalf("unable to retrieve range from document: %v", err)
  }

  fmt.Printf("Selected major dimension=%v, range=%v\n", val.MajorDimension, val.Range)
  for _, row := range val.Values {
    fmt.Println(row)
  }
}

Note the docId passed to the sheets package; this is the path segment in your spreadsheet's URL following the /d/. In this example, I'm using a test sheet I've created.

Important: unless your sheet is world-readable, your service account won't be able to access it. Here the account's email comes in handy; you can take it from the service account's GCP IAM page (Details tab), and give this email permissions to the sheet. This way you can have the program processing a private sheet that only you have access to.

OAuth

Another way to achieve what we want is with OAuth. This also requires a bit of setup in your project's GCP console. Follow the Go quickstart docs for that. Our sample assumes you've saved the credentials.json file somewhere locally and will pass it through the -credfile flag. Unlike the quickstart, it handles all the token exchange process automatically without having to ask you to copy a code from a web page. You still have to authenticate the first time you run it, of course.

The full code of the sample is available on GitHub; while the auth part is different, the actual sheets processing code is identical to the service account sample.

For an overview of the OAuth protocol, see my earlier post.

P.S. ADC

Initially, I had trouble accessing the sheet using ADC (Application Default Credentials), but following a HN comment on this post, I was motivated to try again and it worked. I may have mixed up my auth JSON files previously, because the code is identical to what I've originally tried. In any case, the code is available on GitHub along with the other options. Depending on the exact use case, ADC may be simpler than using a service account (though IMHO the service account is a more "reliable" method across machines because its configuration is more explicit - less is happening under the hood).

Sign in with Google in Go

2024-01-13T06:33:00-08:00

This post provides some code samples for implementing a "Sign-in with Google" flow for your web application in Go. For an overview of auth/authz and the OAuth protocol, please refer to my earlier post about Sign-in with GitHub.

Sign-in with Google has existed in one form or another for many years, and the technical approach to it evolved over time. I will start by presenting the currently recommended way - and the one that will feel most familiar to users - and will then mention a slightly more complicated and flexible alternative.

Using Google Identity Service (GIS)

The currently recommended way to implement sign-in with Google is using the Google Identity Service client library. There's a lot of documentation on that page, and it's worth reading through.

Using this approach, we'll get a standard looking button:

When clicked, a popup opens with a standard-looking Google account selection:

There's also an option for "one tap" which surfaces the currently logged in Google user in an in-page popup to click (this also works well on mobile!)

To get started, you'll need to register an application at https://console.cloud.google.com/apis/credentials and obtain a client ID and secret (the secret isn't used for this approach, but is needed for the next one).

We'll use a Google-provided JS client library called GSI which we include in our web page; it implements the actual button and all the client-side handling; here's a snippet from our Go server for this sample:

// This should be taken from https://console.cloud.google.com/apis/credentials
var GoogleClientID = os.Getenv("GOOGLE_CLIENT_ID")

const servingSchema = "http://"
const servingAddress = "localhost:8080"
const callbackPath = "/google/callback"

var rootHtmlTemplate = template.Must(template.New("root").Parse(`
<!DOCTYPE html>
<html>
<body>
    <script src="https://accounts.google.com/gsi/client" async></script>

    <h1>Welcome to this web app!</h1>
    <p>Let's sign in with Google:</p>
    <div
        id="g_id_onload"
        data-client_id="{{.ClientID}}"
        data-login_uri="{{.CallbackUrl}}">
    </div>
    <div
        class="g_id_signin"
        data-type="standard"
        data-theme="filled_blue"
        data-text="sign_in_with"
        data-shape="rectangular"
        data-width="200"
        data-logo_alignment="left">
    </div>
</body>
</html>
`))

func main() {
  if len(GoogleClientID) == 0 {
    log.Fatal("Set GOOGLE_CLIENT_ID env var")
  }

  mux := http.NewServeMux()
  mux.HandleFunc("/", rootHandler)
  mux.HandleFunc(callbackPath, callbackHandler)

  log.Printf("Listening on: %s%s\n", servingSchema, servingAddress)
  log.Panic(http.ListenAndServe(servingAddress, mux))
}

func rootHandler(w http.ResponseWriter, req *http.Request) {
  err := rootHtmlTemplate.Execute(w, map[string]string{
    "CallbackUrl": servingSchema + servingAddress + callbackPath,
    "ClientID":    GoogleClientID,
  })
  if err != nil {
    panic(err)
  }
}

Note the configuration on the <div> element for the button; there's a ton of potential customization there - the docs explain these in detail.

We serve this page on the root ("/") handler of our local web server, and also register a callback URL which the Google auth server will redirect to once it confirms the user's identity [1]. When it does, it sends the logged-in user's information as a JSON Web Token (JWT); our server-side application should verify the validity of the token and then it's ready to accept the signed-in user.

We'll be using the google.golang.org/api/idtoken package for token verification (this comes from the official GCP client library for Go); here's our callback:

func callbackHandler(w http.ResponseWriter, req *http.Request) {
  defer req.Body.Close()

  if err := req.ParseForm(); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  // The following steps follow
  // https://developers.google.com/identity/gsi/web/guides/verify-google-id-token
  //
  // 1. Verify the CSRF token, which uses the double-submit-cookie pattern and
  //    is added both as a cookie value and post body.
  token, err := req.Cookie("g_csrf_token")
  if err != nil {
    http.Error(w, "token not found", http.StatusBadRequest)
    return
  }

  bodyToken := req.FormValue("g_csrf_token")
  if token.Value != bodyToken {
    http.Error(w, "token mismatch", http.StatusBadRequest)
  }

  // 2. Verify the ID token, which is returned in the `credential` field.
  //    We use the idtoken package for this. `audience` is our client ID.
  ctx := context.Background()
  validator, err := idtoken.NewValidator(ctx)
  if err != nil {
    panic(err)
  }
  credential := req.FormValue("credential")
  payload, err := validator.Validate(ctx, credential, GoogleClientID)
  if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
  }

  // 3. Once the token's validity is confirmed, we can use the user identifying
  //    information in the Google ID token.
  for k, v := range payload.Claims {
    fmt.Printf("%v: %v\n", k, v)
  }
}

The numbered comments follow the documentation step by step, so it should be easy to decipher. The end result is dumping some information about the user from the token; specifically, the Google email and registered name will be provided. Once you have that, you know who the user is and that they have an actual Google account.

This is it! I like how all the complicated front-end details are handled by Google's own JS library; there's a lot of potential nuance there with one-tap and automatic sign-in, different UI requirements for desktop browsers and mobile, etc. If you're developing a standard web-app, this definitely seems like the way to go.

Using OpenID Connect

OpenID Connect is an authentication protocol built on top of OAuth 2. The Google documentation for integrating it into your applications is here.

This is a more generic approach than using GIS - you're not beholden to use specific JS or HTML elements, but can integrate it into your flow in whatever way you wish - similarly to the GitHub approach outlined in my last post. I won't paste the code here - but it's well documented and follows the official docs. I have two separate samples:

Using the go-oidc package: the full code is here.
Using the gologin package: the full code is here. You may recognize the gologin package from my previous post on GitHub sign-in; indeed, the code delta between the two options is very small!

[1]	Please refer to my description of OAuth to understand the redirection flow. This Google sign-in flow still relies on OAuth underneath.

Sign in with GitHub in Go

2023-12-09T14:15:00-08:00

It's common to see web applications that let you log in through third-party services. "Sign in with Google" is particularly popular; on developer-oriented websites, "Sign in with GitHub" also pops up quite a bit. In this post, I want to briefly explore OAuth - the technology that enables these delegated logins, and present some ways to integrate GitHub login in your Go service. Signing in through Google is covered in a separate post.

A note about authentication terminology:

Authentication (authn): is the process of verifying the identity of a user or entity. It answers the question, "Who are you?", typically through credentials like usernames and passwords, 2FA etc.
Authorization (authz): is the process of determining what permissions an authenticated user has with a given service (e.g. Editor, Commenter or Viewer on Google Documents).

This post is about authn, though GitHub really provides a more general authz mechanism. In GitHub, when you attempt to use OAuth login, you ask for specific permissions (called "scopes") ahead of time; so a user authentication process combines authn (does this user have a valid GitHub account?) with authz (can this app get the following permissions to the user's account?)

We try to focus only on authn though, not asking GitHub for any particular permissions other than verifying that a user has an account and getting some basic user information (email) that can be used to uniquely identify the user in our application.

This post provides code samples for accomplishing this task in three ways:

Using nothing but the Go standard library
Using a semi-standard package for handling some of the OAuth minutiae, saving some code
Using a third-party package for handling more of the process, saving even more code

A brief overview of OAuth

OAuth is an authorization standard; currently in version 2.0, it's formally described in RFC 6750. In this post I'll only provide a brief introduction to the standard in the context of the GitHub authentication flow I'm describing; I recommend reading more - start with the Wikipedia page and follow links from there as needed.

Here's a useful diagram describing the OAuth 2.0 (it's taken from this post by Digital Ocean, which is also a good introduction to the subject):

When we want to provide a "Sign-in with GitHub" in our application, the actors involved in the diagram are:

Application (or client): is our application - a web app that wants to allow users to log in with their GitHub account rather than (or in addition to) implementing its own authentication flow.
User: wants to log into our app, and has a GitHub account.
User-agent: the user's web browser
Auth Server: the GitHub OAuth authorization server

The steps involved are:

The user visits the application's website and chooses to log in with GitHub credentials. The application redirects the user to the GitHub auth server which notifies them that "Application FOO" wants them to log in.
The user logs into GitHub; note that this is happening on GitHub's website, in a secured HTTPS session vs. GitHub's server. The user enters their email, password and 2FA if required.
If the login is successful, the auth server redirects the user's browser back to the application, and provides the application with a temporary code it can use to ask for access tokens.
The application uses the temporary code from GitHub along with a secret it has pre-registered with GitHub to request an access token to the user's account.
GitHub checks that everything is kosher and provides an access token to the application.

From this stage on, the application can use the token to query the GitHub API on behalf of the logged-in user, based on the permissions (or "scopes") the user agreed to provide to the application [1]. For logging in, the application just needs to know what the user's email address is.

One important aspect to understand about OAuth for web applications is that it leverages HTTP redirection (status code 301) to route the user between the different servers involved. This is important for user control, security and also convenience (for example, it allows us to register localhost endpoints with GitHub for testing). The way it works is that in step (1) when our application sends the user to GitHub, it adds a redirection URL for GitHub in a request parameter; GitHub uses this to redirect the user back to our page in step (3). We'll see this in action soon.

Sample 1: GitHub OAuth flow using raw stdlib

In our first sample, we're not going to be using any 3rd party packages; instead, we implement a complete GitHub auth flow using the Go standard library.

This sample follows the GitHub documentation: Authorizing OAuth apps, as described in the "Web application" flow [2]. The steps outlined on that page are directly mapped to steps in my sample Go program. If you want to follow along, make sure to register an application with GitHub, set a compatible callback path and write down your client ID and client secret; my code expects these to be in the env vars GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET, respectively.

Our application starts by registering some HTTP routes:

// These should be taken from your GitHub application settings
// at https://github.com/settings/developers
var GithubClientID = os.Getenv("GITHUB_CLIENT_ID")
var GithubClientSecret = os.Getenv("GITHUB_CLIENT_SECRET")

func main() {
  if len(GithubClientID) == 0 || len(GithubClientSecret) == 0 {
    log.Fatal("Set GITHUB_CLIENT_* env vars")
  }

  http.HandleFunc("/", rootHandler)
  http.HandleFunc("/login/", githubLoginHandler)
  http.HandleFunc("/github/callback/", githubCallbackHandler)

  addr := "localhost:8080"
  fmt.Printf("Listening on: http://%s\n", addr)
  log.Panic(http.ListenAndServe(addr, nil))
}

When the server runs, the user is expected to visit the root route /, where they're presented with a link to "Log in with GitHub". This is done with the following handler:

const rootHTML = `
<h1>My web app</h1>
<p>Using raw HTTP OAuth 2.0</p>
<p>You can log into this app with your GitHub credentials:</p>
<p><a href="/login/">Log in with GitHub</a></p>
`

func rootHandler(w http.ResponseWriter, r *http.Request) {
  fmt.Fprint(w, rootHTML)
}

Once the user clicks the "log in" link, they're taken to the /login/ handler:

func githubLoginHandler(w http.ResponseWriter, r *http.Request) {
  // Step 1: Request a user's GitHub identity
  //
  // ... by redirecting the user's browser to a GitHub login endpoint. We're not
  // setting redirect_uri, leaving it to GitHub to use the default we set for
  // this application: /github/callback
  // We're also not asking for any specific scope, because we only need access
  // to the user's public information to know that the user is really logged in.
  //
  // We're setting a random state cookie for the client to return
  // to us when the call comes back, to prevent CSRF per
  // section 10.12 of https://www.rfc-editor.org/rfc/rfc6749.html
  state, err := randString(16)
  if err != nil {
    panic(err)
  }

  c := &http.Cookie{
    Name:     "state",
    Value:    state,
    Path:     "/",
    MaxAge:   int(time.Hour.Seconds()),
    Secure:   r.TLS != nil,
    HttpOnly: true,
  }
  http.SetCookie(w, c)

  redirectURL := fmt.Sprintf("https://github.com/login/oauth/authorize?client_id=%s&state=%s", GithubClientID, state)
  http.Redirect(w, r, redirectURL, 301)
}

The comment explains how this maps to step 1 in the documented GitHub flow. The note about redirect_url is important; typically the auth server expects a redirect_url path. However, it should also be configured when you register the application. Here I opt for using the default configuration and am not passing redirect_url explicitly.

This code implements CSRF protection by generating a random string, storing it in a cookie for this user's session and sending it with the state URL parameter to GitHub. When GitHub calls our redirect URL, it will attach this state - we can thus verify the request is legit and not a malicious attack.

At this point the user is presented with a familiar "sign-in to GitHub" screen on GitHub's website:

The user verifies their credentials vs. GitHub; then GitHub redirects the user's browser back to the redirect URL provided (or configured in the GitHub app settings). In our case, the redirect goes to http://localhost:8080/github/callback/ [3], which is mapped to githubCallbackHandler. Here's the first part of this function, implementing step 2 from the documented GitHub flow:

func githubCallbackHandler(w http.ResponseWriter, r *http.Request) {
  // Step 2: Users are redirected back to your site by GitHub
  //
  // The user is authenticated w/ GitHub by this point, and GH provides us
  // a temporary code we can exchange for an access token using the app's
  // full credentials.
  //
  // Start by checking the state returned by GitHub matches what
  // we've stored in the cookie.
  state, err := r.Cookie("state")
  if err != nil {
    http.Error(w, "state not found", http.StatusBadRequest)
    return
  }
  if r.URL.Query().Get("state") != state.Value {
    http.Error(w, "state did not match", http.StatusBadRequest)
    return
  }

  // We use the code, alongside our client ID and secret to ask GH for an
  // access token to the API.
  code := r.URL.Query().Get("code")
  requestBodyMap := map[string]string{
    "client_id":     GithubClientID,
    "client_secret": GithubClientSecret,
    "code":          code,
  }
  requestJSON, err := json.Marshal(requestBodyMap)
  if err != nil {
    panic(err)
  }

  req, err := http.NewRequest("POST", "https://github.com/login/oauth/access_token", bytes.NewBuffer(requestJSON))
  if err != nil {
    panic(err)
  }
  req.Header.Set("Content-Type", "application/json")
  req.Header.Set("Accept", "application/json")

  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    http.Error(w, "unable to connect to access_token endpoint", http.StatusInternalServerError)
    return
  }
  respbody, _ := io.ReadAll(resp.Body)

  // Represents the response received from Github
  var ghresp struct {
    AccessToken string `json:"access_token"`
    TokenType   string `json:"token_type"`
    Scope       string `json:"scope"`
  }
  json.Unmarshal(respbody, &ghresp)

  // {...}

The function does just what it says on the box: after verifying the CSRF token, it provides the code it got from GitHub back to GitHub's OAuth access token endpoint, along with a secret shared only by the application and GitHub (it's in the app settings). In return it gets a token that can be used as a bearer token for HTTP auth when accessing the GitHub API on behalf of the logged-in user. Here's the rest of the code:

func githubCallbackHandler(w http.ResponseWriter, r *http.Request) {
  // {...}

  // Step 3: Use the access token to access the API
  //
  // With the access token in hand, we can access the GitHub API on behalf
  // of the user. Since we didn't provide a scope, we only get access to
  // the user's public information.
  userInfo := getGitHubUserInfo(ghresp.AccessToken)

  w.Header().Set("Content-type", "application/json")
  fmt.Fprint(w, string(userInfo))
}

// getGitHubUserInfo queries GitHub's user API for information about the
// authorized user, given the access token received earlier.
func getGitHubUserInfo(accessToken string) string {
  // Query the GH API for user info
  req, err := http.NewRequest("GET", "https://api.github.com/user", nil)
  if err != nil {
    panic(err)
  }
  req.Header.Set("Authorization", "Bearer "+accessToken)

  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    panic(err)
  }

  respbody, _ := io.ReadAll(resp.Body)
  return string(respbody)
}

The end result of this is dumping a JSON map with the logged-in user's information to output. This is just an example, of course. A real web application would now use the user's email as a unique ID to look up things in its own DB - it knows the user is authenticated now!

Sample 2: using the x/oauth2 package

The second sample uses the golang.org/x/oauth2 package to take care of some of the OAuth details instead of doing everything manually. Much of the code remains the same - I just want to highlight some places where x/oauth2 is used instead of writing code from scratch. First, we create a configuration struct in our main function:

conf := &oauth2.Config{
        ClientID:     GithubClientID,
        ClientSecret: GithubClientSecret,
        Scopes:       []string{},
        Endpoint:     github.Endpoint,
}

Here github refers to the x/oauth2/github subpackage, which holds some constants useful for GitHub OAuth. Specifically, github.Endpoint is an alias to:

var GitHub = oauth2.Endpoint{
  AuthURL:       "https://github.com/login/oauth/authorize",
  TokenURL:      "https://github.com/login/oauth/access_token",
  DeviceAuthURL: "https://github.com/login/device/code",
}

So our code needn't have these paths hard-coded.

In githubLoginHandler, we can use the AuthCodeURL method instead of constructing the URL manually.

Finally, in githubCallbackHandler we get help in two ways:

code := r.URL.Query().Get("code")
tok, err := lf.conf.Exchange(context.Background(), code) // 1: token exchange
if err != nil {
        log.Fatal(err)
}

// This client will have a bearer token to access the GitHub API on
// the user's behalf.
client := lf.conf.Client(context.Background(), tok)     // 2: client with token
resp, err := client.Get("https://api.github.com/user")
if err != nil {
        panic(err)
}
respbody, _ := io.ReadAll(resp.Body)
userInfo := string(respbody)

w.Header().Set("Content-type", "application/json")
fmt.Fprint(w, string(userInfo))

First, we can call Exchange to ask x/oauth2 to perform the "code for token" exchange step. Second, we get an HTTP client with the bearer token already configured and don't have to do it manually. The resulting code is somewhat shorter and has fewer hard-coded operations, delegating some of the work to the x/oauth2 package.

Sample 3: using the gologin package

Finally, our third sample uses the third-party gologin package. gologin implements helpers for logging into various services like GitHub, Google and Twitter. It encapsulates even more functionality. Here's the relevant part of the main function in this sample:

conf := &oauth2.Config{
        ClientID:     GithubClientID,
        ClientSecret: GithubClientSecret,
        Scopes:       []string{},
        Endpoint:     oauth2github.Endpoint,
}

// gologin has a default cookie configuration for debug deployments (no TLS).
cookieConf := gologin.DebugOnlyCookieConfig
http.HandleFunc("/", rootHandler)

loginHandler := github.LoginHandler(conf, nil)
http.Handle("/login/", github.StateHandler(cookieConf, loginHandler))

callbackHandler := github.CallbackHandler(conf, http.HandlerFunc(githubCallbackHandler), nil)
http.Handle(callbackPath, github.StateHandler(cookieConf, callbackHandler))

gologin relies upon the x/oauth2 package and uses its Config. The rest of the code registers the HTTP handlers, and it has two interesting properties:

gologin handles the CSRF cookie management for us, using its StateHandler middleware.
It also handles the token exchange with GitHub automatically, by exposing a CallbackHandler middleware that wraps our githubCallbackHandler handler.

Our handler gets user information through the request context:

func githubCallbackHandler(w http.ResponseWriter, r *http.Request) {
  ctx := r.Context()
  githubUser, err := github.UserFromContext(ctx)
  if err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    return
  }

  w.Header().Set("Content-type", "application/json")
  buf, _ := json.Marshal(githubUser)
  fmt.Fprint(w, string(buf))
}

That's about it! gologin encapsulates the rest of the functionality through its middleware, enabling GitHub logins in only a handful of lines of code.

Which approach to choose?

All the samples in this post accomplish the same goal - they let a web app delegate its user authentication to GitHub. The samples are presented in increasing level of encapsulation and external dependency - from a completely raw HTTP-based approach using only the stdlib, to using a package that does almost all the work for us.

How to choose between these is a question every project may answer differently, based on their stage, resources and approach to dependencies. I'd probably go for the second approach - using x/oauth2 - since it balances dependencies with utility.

The x/oauth2 package is "semi-official", maintained by some members of Go team and other Go users at Google, and it's helpful without being magical. That said, the gologin package also looks pretty solid, so that could be a good option too if your tolerance to third-party dependencies is higher.

Code

The full code for this post is available on GitHub.

[1]

This is an artifact of using GitHub's authz flow for authn. Generally, this flow allows applications (think CI systems) to have broader access to the user's GitHub account in order to render some useful service to the user. In our case we only use the flow for authn, so all we need is "basic account access", to know that the user indeed has valid access to a GitHub account and the public details of this account.

[2]	Kudos to GitHub for the solid documentation they have on this topic. They also have a complete sample in Ruby which is similar to my Sample 1.

[3] This is a good time to admire the choice OAuth made to use redirects for this step. Consider the following questions: without redirects, how would you tell GitHub to dial to some arbitrary path on your localhost? Also, without redirects how could this request use the user's browser cookies to enable easy storage of CSRF tokens?

Reverse proxying a sub-domain via Apache

2023-01-21T06:26:00-08:00

Suppose you have a domain that hosts your website: domain.com, and the website is served with the venerable Apache HTTP server. Suppose, also, that you want to run some backend application on the same domain, perhaps using a sub-domain like sub.domain.com. Running an application on a non-standard port (not 80 or 443) is not a problem, but what it you need it to run on port 80? Apache occupies port 80 in order to serve domain.com, so at least on the surface this seems like a problem.

This post talks about how to make it work using the reverse-proxying capabilities of Apache. It assumes you control a virtual machine that has a top-level domain like domain.com mapped to it, and that the machine runs Linux.

Setting up Apache as a proxy with mod_proxy

If you need to brush up on proxy concepts, consider reading this series of posts first.

Assuming Apache is already installed and running on the server, you'll first have to enable the proxy module and restart the service:

$ sudo a2enmod proxy proxy_http
$ sudo systemctl restart apache2

Sub-domains typically have their own configuration file in /etc/apache2/sites-available. Create a new configuration file in that directory, named sub.domain.com.conf or some such; here's what should be in it (adjust as needed):

<VirtualHost *:80>
        ProxyPreserveHost On
        ProxyPass / http://127.0.0.1:5000/
        ProxyPassReverse / http://127.0.0.1:5000/

        ServerName sub.domain.com
        ServerAdmin your@email.com

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

This tells Apache that the sub.domain.com route should be proxied to a service running locally on port 5000; naturally, the service address can have a different port or run on a different domain altogether.

Next you'll want to register that configuration with Apache and restart it again:

$ sudo a2ensite sub.domain.com.conf
$ sudo systemctl restart apache2

Running the backend service

Now that Apache is all set up, it's time to run the actual backend service at port 5000. As an example, you can run this simple header debugging server:

$ go run http-server-debug-request-headers.go -addr 127.0.0.1:5000
2023/01/17 01:01:20 Starting server on 127.0.0.1:5000

To test that it runs properly, in a separate terminal (on the same machine!) let's run curl:

$ curl 127.0.0.1:5000/headers
hello /headers

And looking at the terminal where the server is running, you should see some useful logging:

2023/01/17 01:02:50 127.0.0.1:42406   GET     /headers        Host: 127.0.0.1:5000
User-Agent: curl/7.81.0
Accept: */*

If you've followed all the steps in this and the previous session, it should work via the sub-domain now (from any machine):

$ curl http://sub.domain.com/headers
hello /headers

Apache listens on port 80 for domain.com, and when it sees requests to sub.domain.com, it proxies them to the server running on port 5000 on the same machine.

If this doesn't work for you, take a careful look at the Apache logs - both the error log and the access log may be useful.

Bonus: TLS with Let's Encrypt

If your server is set up to serve domain.com via TLS using Let's Encrypt, I have good news for you -- it will just work for sub.domain.com as well!

Presumably you've set up Let's Encrypt certificates using certbot. Since we've now added an additional Apache configuration (sub.domain.com.conf), we should run certbot again:

$ sudo certbot --apache

And carefully follow the on-screen instructions. certbot should detect there's a new sub-domain to get a certificate for; if everything goes as expected, it succeeds and from that point on you should be able to access the backend server via HTTPS:

$ curl https://sub.domain.com/headers
hello /headers

Note that the backend Go server serves HTTP; the reverse proxy (Apache) terminates the TLS connection and passes HTTP to the backend server. This is a fairly common way to structure backends. While the backend server serves unencrypted traffic, it's not actually accessible from outside the machine (port 5000 is unlikely to be exposed). The only way to access it is via the reverse-proxy on sub.domain.com, which can use TLS if needed.

I was wondering how this works. certbot uses the HTTP challenge with Let's Encrypt, wherein it's asked to serve a special file on a special path (typically something like .well-known/acme-challenge) to prove to Let's Encrypt that it controls the domain. But here all requests get forwarded to the backend server...

After scratching my head for a minute I found the answer in certbot's logs, where it honestly explains its tricky ways. It turns out it adds a RewriteRule to our sub.domain.com.conf file for the duration of the Let's Encrypt handshake, sending any requests starting with .well-known/acme-challenge to a known disk location it controls. After all is done, it quietly removes these rules from the configuration file.

Review of the Coursera Machine Learning course

2015-01-28T05:14:00-08:00

I've just finished going through the Machine Learning course on Coursera, and this is a brief review.

The scope of the course is quite broad, covering a lot of topics from both supervised and unsupervised machine learning. "Practical advice" parts are sprinkled throughout the lectures and are extremely useful - how to evaluate your algorithm, how to know what to focus on, practical tips for improving speed and accuracy, etc.

Machine Learning is one of the oldest courses on Coursera, and is extremely polished by now. In the lectures, Andrew Ng foresees the questions that may arise and answers them beforehead. The programming excercises are very well organized with very little detail left to chance. They're quite sizable too. I added extra work to myself by using Python instead of Octave (if you use Octave or Matlab a lot gets set up for you by the code provided with the assignment), but that's probably not the bulk of the code. ~2200 lines of dense numerical code (in throwaway homework mode - barely any comments and few unit tests) is quite a lot.

One thing I liked somewhat less is the lack of mathematical depth. I realize the reason for this is the relatively low barrier of entry they want to set for the course, I'm sure that Stanford students taking the real course are expected to know some linear algebra and probability and the formulae can be developed, rather than just presented as an act of god. This was especially noticeable in the lecture on PCA, where Prof Ng just dropped the eigenvector approach, and even went so far as to provide the exact Octave function to call, without much mathematical background or reasoning. Indeed, it is quite possible to implement the PCA part of the programming exercises without really understanding how PCA works.

Which brings me to a related topic. The course is way too easy to provide a meaningful certificate. The review quizzes are trivial and require very little thought. The programming exercises, while demanding in terms of invested time, aren't going deep either, and don't deviate from the lecture materials - you basically follow very detailed instructions and transcribe formulae into code.

Overall, I enjoyed the course, and I would highly recommend it to anyone interested in getting into machine learning (or, by its new-age name, big data).

Finally, an interesting observation I had while working through the programming assignments in Python (using Numpy, Scipy and Scikit-learn). Domain specific languages (like Matlab/Octave for general numerics and R for statistics) are often hailed as eye-openers because they come equipped with some facilities tailored at scientific programming (like a convenient notation for defining constant matrices), but I don't think this is the right approach. Writing Python with Numpy et. al, some basic things may take a bit more keystrokes to achieve, but eventually you end up with very similar code. Moreover, it's all the same Fortran-written LINPACK running under the hood anyways, so performance is the same.

However, expressing a constant matrix in the most concise way possible is not the end of the story when writing a machine learning system. It's nice to have an actual programming language in your hands, with all that entails - community, libraries, etc. It was curious to notice the impact of this in the spam classification assignment. Tons of textual preprocessing for which Octave is not very well suitable. Need to implement some NLP algorithms, for which Octave has no libraries, etc. In Python it was all a breeze, of course, including importing NLTK to take care of any NLP needs.

I'm not saying that Python is the best tool for every job. But for exploratory scientific programming, it seems like the strongest option out there. Numpy and its kin are very mature, fast and well-supported. Plotting with matplotlib is great (and I heard that the legendary ggplot from R now has Python bindings as well). And when you need to step outside the narrow domain of computations, you have the full programming language with all its support structure at your command.