Hot restart golang server

Keywords: socket Front-end Nginx github

Server-side code often needs to be upgraded. For online system upgrade, the usual method is to ensure that at least one service is available through the front-end load balancing (such as nginx) to upgrade, in turn (gray level).
Another more convenient way is to do hot restart in the application, upgrade the application directly and keep serving.

principle

The principle of hot restart is very simple, but it involves some system calls and the transfer of file handles between parent and child processes.
The process is divided into the following steps:

  1. Monitoring Signal (USR2)
  2. The fork subprocess (using the same startup command) passes the socket file descriptor that the service monitors to the subprocess when it receives the signal
  3. The child process listens to the socket of the parent process, at which point both the parent process and the child process can receive requests.
  4. After the child process starts successfully, the parent process stops receiving new connections and waits for the old connection processing to complete (or timeout)
  5. The parent process exits and the upgrade is complete

details

  • The parent process passes the socket file descriptor to the child process through the command line, or environment variables, etc.
  • The child process starts with the same command line as the parent process, and for golang, overrides the old program with an updated executable
  • server.Shutdown() Elegant Closing Method is a new feature of go1.8
  • The server.Serve(l) method returns immediately upon hutdown, while the Shutdown method blocks until context completes, so the Shutdown method is written in the main goroutine.

Code

package main

import (
    "context"
    "errors"
    "flag"
    "log"
    "net"
    "net/http"
    "os"
    "os/exec"
    "os/signal"
    "syscall"
    "time"
)

var (
    server   *http.Server
    listener net.Listener
    graceful = flag.Bool("graceful", false, "listen on fd open 3 (internal use only)")
)

func handler(w http.ResponseWriter, r *http.Request) {
    time.Sleep(20 * time.Second)
    w.Write([]byte("hello world233333!!!!"))
}

func main() {
    flag.Parse()

    http.HandleFunc("/hello", handler)
    server = &http.Server{Addr: ":9999"}

    var err error
    if *graceful {
        log.Print("main: Listening to existing file descriptor 3.")
        // cmd.ExtraFiles: If non-nil, entry i becomes file descriptor 3+i.
        // when we put socket FD at the first entry, it will always be 3(0+3)
        f := os.NewFile(3, "")
        listener, err = net.FileListener(f)
    } else {
        log.Print("main: Listening on a new file descriptor.")
        listener, err = net.Listen("tcp", server.Addr)
    }

    if err != nil {
        log.Fatalf("listener error: %v", err)
    }

    go func() {
        // server.Shutdown() stops Serve() immediately, thus server.Serve() should not be in main goroutine
        err = server.Serve(listener)
        log.Printf("server.Serve err: %v\n", err)
    }()
    signalHandler()
    log.Printf("signal end")
}

func reload() error {
    tl, ok := listener.(*net.TCPListener)
    if !ok {
        return errors.New("listener is not tcp listener")
    }

    f, err := tl.File()
    if err != nil {
        return err
    }

    args := []string{"-graceful"}
    cmd := exec.Command(os.Args[0], args...)
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    // put socket FD at the first entry
    cmd.ExtraFiles = []*os.File{f}
    return cmd.Start()
}

func signalHandler() {
    ch := make(chan os.Signal, 1)
    signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM, syscall.SIGUSR2)
    for {
        sig := <-ch
        log.Printf("signal: %v", sig)

        // timeout context for shutdown
        ctx, _ := context.WithTimeout(context.Background(), 20*time.Second)
        switch sig {
        case syscall.SIGINT, syscall.SIGTERM:
            // stop
            log.Printf("stop")
            signal.Stop(ch)
            server.Shutdown(ctx)
            log.Printf("graceful shutdown")
            return
        case syscall.SIGUSR2:
            // reload
            log.Printf("reload")
            err := reload()
            if err != nil {
                log.Fatalf("graceful restart error: %v", err)
            }
            server.Shutdown(ctx)
            log.Printf("graceful reload")
            return
        }
    }
}

references

Posted by s4salman on Thu, 03 Jan 2019 19:12:09 -0800

Hot Keywords

©2024 Programmer Group