Go Fundamentals - Sample

Table of Contents

Chapter 14.12: Using the FS Interface

Previously, we had been using filepath.WalkDir to walk the directory tree. This worked by walking the file system directly. As a result, we have to make sure that the file system is in the correct state before we can use it. As we have seen, this can cause a lot of setup work to be done. Either we have to keep a completely different folders for each test scenario we have, or we have to create all of the files and folders at the start of the test before we begin. This is a lot of work, and is often error prone.

$ go doc fs.WalkDir

package fs // import "io/fs"

func WalkDir(fsys FS, root string, fn WalkDirFunc) error
    WalkDir walks the file tree rooted at root, calling fn for each file or
    directory in the tree, including root.

    All errors that arise visiting files and directories are filtered by fn:
    see the fs.WalkDirFunc documentation for details.

    The files are walked in lexical order, which makes the output deterministic
    but requires WalkDir to read an entire directory into memory before
    proceeding to walk that directory.

    WalkDir does not follow symbolic links found in directories, but if root
    itself is a symbolic link, its target will be walked.

--------------------------------------------------------------------------------
Go Version: go1.19

Listing 14.1: The fs.WalkDir function.

If we use the fs.WalkDir function, Listing 14.1, instead, which takes, as its first argument, a fs.FS implementation. Instead of walking the file system the fs.FS is walked instead.

func Walk(cab fs.FS) ([]string, error) {
	var entries []string

	err := fs.WalkDir(cab, ".", func(path string, d fs.DirEntry, err error) error {

		// if there was an error, return it
		// if there is an error, it is most likely
		// because an error was an encoutered trying
		// to read the top level directory
		if err != nil {
			return err
		}

		// if the entry is a directory, handle it
		if d.IsDir() {

			// name of the file or directory
			name := d.Name()

			// if the directory is a dot return nil
			// this may be the root directory
			if name == "." || name == ".." {
				return nil
			}

			// if the directory name is "testdata"
			// or it starts with "."
			// or it starts with "_"
			// then return filepath.SkipDir
			if name == "testdata" || strings.HasPrefix(name, ".") || strings.HasPrefix(name, "_") {
				return fs.SkipDir
			}

			return nil
		}

		// append the entry to the list
		entries = append(entries, path)

		// return nil to tell walk to continue
		return nil
	})

	return entries, err
}
Listing 14.2: Using the fs.WalkDir function.

In Listing 14.2 we can continue to use the same code inside of the fs.WalkFunc function as we did before. Our test code only needs two small changes now to make it work. The first is we an implementation of the fs.FS interface to pass to the Walk function. For now, we can use the os.DirFS function, Listing 14.3. The implementation will be backed directly by the file system.

$ go doc os.DirFS

package os // import "os"

func DirFS(dir string) fs.FS
    DirFS returns a file system (an fs.FS) for the tree of files rooted at the
    directory dir.

    Note that DirFS("/prefix") only guarantees that the Open calls
    it makes to the operating system will begin with "/prefix":
    DirFS("/prefix").Open("file") is the same as os.Open("/prefix/file").
    So if /prefix/file is a symbolic link pointing outside the /prefix tree,
    then using DirFS does not stop the access any more than using os.Open does.
    Additionally, the root of the fs.FS returned for a relative path,
    DirFS("prefix"), will be affected by later calls to Chdir. DirFS is
    therefore not a general substitute for a chroot-style security mechanism
    when the directory tree contains arbitrary content.

    The result implements fs.StatFS.

--------------------------------------------------------------------------------
Go Version: go1.19

Listing 14.3: The os.DirFS function.

File Paths

The other change we need to make is the expected paths. Before, we were expecting paths such as /data/a.txt returned from our Walk function. However, when working with fs.FS implementations, the paths returned are relative to the root of the implementation. In this case, we are using os.DirFS("data") to create the fs.FS implementation. This places data at the root of the file system implementation and paths returned from the Walk function will be relative to this root.

Note: Paths are expected to use / as the path separator, regardless of the operating system.

We need to update our test code to for the relative paths, a.txt, to be returned instead of /data/a.txt.

func Test_Walk(t *testing.T) {
	t.Parallel()

	exp := []string{
		"a.txt",
		"b.txt",
		"e/f/g.txt",
		"e/f/h.txt",
		"e/j.txt",
	}

	cab := os.DirFS("data")

	act, err := Walk(cab)
	if err != nil {
		t.Fatal(err)
	}

	es := strings.Join(exp, ", ")
	as := strings.Join(act, ", ")

	if es != as {
		t.Fatalf("expected %s, got %s", es, as)
	}
}

$ go test -v

=== RUN   Test_Walk
=== PAUSE Test_Walk
=== CONT  Test_Walk
--- PASS: Test_Walk (0.00s)
PASS
ok  	demo	0.537s

--------------------------------------------------------------------------------
Go Version: go1.19
Listing 14.4: Testing the Walk function.

As we can see from the test output in Listing 14.4, we have successfully updated our code to use the fs.FS interface.


Subscribe to our newsletter