FancyIndex - Better directory listing Published the 2017-11-21 Recently, I got my hands on a small server, with a big storage space. I decided to use it as a lightweight archive file server, accessible over HTTPS. So, let's try to do that today! ## Searching available tools I already know about a PHP/Js web-based file browser called [h5ai](https://larsjung.de/h5ai/). The design is quite nice but having a JS frontend is a huge constraint (remember, lightweight was a goal) and having to install and configure an entire PHP backend is bothersome. If we try to check a bit the alternatives, we almost only get huge webapps, r/w file browsers etc., which is clearly too much for this need. I chose NGINX as the web server, to easily handle HTTP/2, SSL etc. So, let's check out the auto-indexing system of NGINX! It's generally the preferred way to do a web Read-only FS, and really fills its job, as it's the most minimalistic, simple and direct one available. Still, the blank design and the contrast hurts the eyes, and the listing is a bit too small. Let's try to see if NGINX have a way to customize that page! The only thing I could find that would be "basic" was the [Fancy index](https://www.nginx.com/resources/wiki/modules/fancy_index/) extension, but it doesn't look maintained, up to date and we don't even know if it's compatible, and if so, for how long. Sooo, that's a no-no. During this search, I thought about the compatibility issues: we don't want to tightly couple the web server and the directory listing printing, as, the day we'd move away from NGINX, we would have up to nothing new to handle, no matter the server. For that reason, and following everything we found, let's roll our own! ## The requirements Let's make a small specification for what the project solution will do. ``` This tool is a small and lightweight server that prints a directory's content. It uses the URL path and a configured base path to match directories. If the directory cannot be found, it will return a custom 404 page. The base path must exist and point to a valid folder. If not, at startup, the server must crash with a clear error message showing that it's not valid. ``` Sounds good for a beginning. Hhm... and what about the internal listing/file access logic? Since this tool is made to run behind a reverse-proxy such as NGINX or Apache, as it only fills the role of directory-listing, the simplest configuration would follow the next logic: - Is the pointed path a valid file under the host's document root? yes -> serve the file, no -> continue - Proxy_pass to the server. Looks perfect ! Let's add that to the spec. Now that we have a clearer view on what to do and how it should act, let's think of what we actually want to use... I eyed the Go language for some time, especially since: - it's a compiled language - the stdlib ships every tool to have a basic functional web server Still, I never got the time to get a hang of it. Well, a simple project is always the best way to start learning a language! ## Let's go ! [webtut]: https://golang.org/doc/articles/wiki/ By a quick search, we find [this tutorial][webtut], directly from the GoLang wiki. If we look at what's proposed in this tutorial, we can see that, well... there's everything we would need to start learning! > Covered in this tutorial: > > - Creating a data structure with load and save methods > - Using the net/http package to build web applications > - Using the html/template package to process HTML templates > - Using the regexp package to validate user input > - Using closures > > *From [the "Writing Web Applications"][webtut] tutorial.* Now... For each request, the logic would be the following: - Grabbing the request URI - Constructing real system path by coupling a defined base path and the URI - If the resulting path doesn't point to a valid directory, return a 404 - If the resulting path points to a valid directory, lists its content and returns a formatted list The first thing we could do, to simplify our work but also to start learning about Go, is directory listing. For now, let's try to write a function that lists the content of the working directory (`pwd`). the [`ioutil` standard lib](https://golang.org/pkg/io/ioutil/) have a `ReadDir(string) ([]os.FileInfo, error)` that is described as > ReadDir reads the directory named by dirname and returns a list of directory entries sorted by filename. Looks perfect! ```go func handler() { dirs, err := ioutil.ReadDir(".") if err != nil { panic("Unable to read directory") } } ``` Now that we have a way to cleanly list a directory content (and return a 404 if `error == nil`), we just need to iterate through the resulting array and just grab the bits of informations that we want. Those informations are, separately, the folders and everything else: we want to be able to display all folders first. Let's make a small struct to handle our result set! > Here, Files is used to describe every reachable entity that is not a folder, > so symbolic links etc. ```go type Res struct { Path string Folders []string // For now, we'll want an array of folder names Files []string // For now, we'll want an array of file names } func handler() { res := &Res{ Path: ".", Folders: []string{}, Files: []string{}, } dirs, err := ioutil.ReadDir(res.Path) if err != nil { panic("Unable to read directory") } } ``` That will hold everything together nicely. We then can simply add a function that will iterate on each directory entry in dirs and, if it's a directory, add it to the Folders array, but else in the Files array. ```go type Res struct { Path string Folders []string // For now, we'll want an array of folder names Files []string // For now, we'll want an array of file names } func handler() { res := &Res{ Path: ".", Folders: []string{}, Files: []string{}, } dirs, err := ioutil.ReadDir(res.Path) if err != nil { panic("Unable to read directory") } for _, f := range dirs { name := f.Name() if f.IsDir() { // If the directory is not named as "myDirectory/", add that final slash if !strings.HasSuffix(name, "/") { name += "/" } res.Folders = append(res.Folders, name) } else { res.Files = append(res.Files, name) } } } ``` Simple, and that works ! Now, letting this rest a bit, let's try to play a bit with the HTTP server stuff and see what we can use. If we look at the given example in the [tutorial][webtut], we have this code to start with: ```go package main import ( "fmt" "net/http" ) func handler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:]) } func main() { http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } ``` Okay, so I guess `handler` is the actual request handler, as it have both a `http.ResponseWriter` struct and a `http.Request` struct (over a pointer). By looking at the only instruction in the handler body, it looks like it's printing in a flux named `w`, which is our `http.ResponseWriter` variable, so I guess it's here I'd want to look into for organizing my directory listing. But something catches the eye: the value passed in the print formatter: `r.URL.Path[1:]`. After investigation ([Request](https://golang.org/pkg/net/http/#Request), [url](https://golang.org/src/net/url/url.go?s=8841:9357#L318)), it seems like the entire URI is stored as an array of URL bits which, together, compose the URI. That's neat, we'll keep that in memory, it will surely come in handy later! Continuing our code reading, I see that this handler is effectively "mounted" to the route "/" (`http.HandleFunc("/", handler)`). Nothing weird here. If we try that example, something can immediately be noticed: whatever we type as a URI path ends up printed in the answer. That means two things: 1. Every request that does not find a more precise or previously defined path is handled by the "/" handler. 2. We have, just here, 90% of our request handling! (Remember, the only thing we want is the path the user is trying to access). I'll overlook the bits about the base path configuration and such. Instead, we'll concentrate on the last thing we actually want: templates. We want to be able to stylise as we want our directory listing, without a pain. Simple answer: HTML templates. Does our magic tutorial have that? Yes! The only thing we actually need to modify in our code is to add the template loading and change the Fprintf to the template engine's way of sending back content. If we take a look at the given example, ```html

Editing {{.Title}}

``` and the Go handler, ```go func editHandler(w http.ResponseWriter, r *http.Request) { title := r.URL.Path[len("/edit/"):] p, err := loadPage(title) if err != nil { p = &Page{Title: title} } t, _ := template.ParseFiles("edit.html") t.Execute(w, p) } ``` > *Note that `p` is a struct defined a bit before this part, > in the web tutorial.* ```go type Page struct { Title string Body []byte } ``` We can easily adapt our code to use a HTML template ! ```go func handler(w http.ResponseWriter, r *http.Request) { // res := &Res{} t, _ := template.ParseFiles("list.html") t.Execute(w, res) } ``` I could've continued to read the guide further (and I'll actually do so for my next tweakings with Go), but as far as I'm concerned, we have everything we need right now ! So after a bit of prototyping, I can finally come to this source: ```go package main import ( "bytes" "fmt" "html/template" "io/ioutil" "net/http" "os" "strings" "time" config "./config" ) // Res format of a folder's content type Res struct { BasePath string Path string ParentPath string IsNotRoot bool Folders []string // For now, we'll want an array of folder names Files []string // For now, we'll want an array of file names Motd string IsMotdEnabled bool } func parentPath(str string) string { if strings.Compare(str, "/") == 0 { return str } if strings.HasSuffix(str, "/") { str = str[:len(str)-1] } if strings.Count(str, "/") == 1 { return "/" } lio := strings.LastIndex(str, "/") return str[:lio] } func handler(w http.ResponseWriter, r *http.Request) { res := &Res{ BasePath: config.RootFSPath, Path: r.URL.Path, Folders: []string{}, Files: []string{}, Motd: config.Motd, IsMotdEnabled: config.MotdEnabled, } res.ParentPath = parentPath(res.Path) res.IsNotRoot = strings.Compare(res.Path, "/") != 0 // Uniforming the end '/' if !strings.HasSuffix(res.Path, "/") { res.Path += "/" } var path bytes.Buffer path.WriteString(res.BasePath) path.WriteString(res.Path) dirs, err := ioutil.ReadDir(path.String()) if err != nil { t, _ := template.ParseFiles("templates/404.html") t.Execute(w, res) return } for _, f := range dirs { name := f.Name() if f.IsDir() { if !strings.HasSuffix(name, "/") { name += "/" } res.Folders = append(res.Folders, name) } else { res.Files = append(res.Files, name) } } t, _ := template.ParseFiles("templates/list.html") t.Execute(w, res) } func main() { http.HandleFunc("/", handler) ListenURL := os.Getenv("HOST") if len(ListenURL) == 0 { ListenURL = "127.0.0.1:3001" } StartDate := time.Now().String() fmt.Printf( "Server starting at %s.\nListening to incoming requests on \"%s\"\n", StartDate, ListenURL) http.ListenAndServe(ListenURL, nil) } ``` You'll also notice two additional things: - MOTD (which is basically a static text display that can be enabled and customized outside of the template itself). - A custom 404 page, too. This server works great, handles requests blazingly fast and is dead simple to hack with! I guess, as it's still beginner code, that a lot of things could be improved, but as far as I'm concerned, the tool is efficient as-is!