AWS EFS Configuration failure

I wanted to share a quick story about setting up an AWS EFS instance. If you're reading this, I assume you're either in the process of configuring one or have encountered a similar issue. Here's what happened: You successfully set up an EFS, configured access points for it, and mounted it to an EC2 instance or ECS service cluster. You thought everything was going smoothly because you successfully set up NAT networking. However, when you tried to write data to the filesystem, it kept failing with a Permission denied exception.

This happens due to misconfiguration of the access point permissions: the directory created in the EFS doesn't imply permissions for the user trying to reach the data on disk. For example, imagine an access point that was made with the configuration to be owned by the uid 1000 and gid 1000 with the permissions set to 755:

access_points = {
  data = {
    posix_user = {
      uid = 1000
      gid = 1000
    }
    root_directory = {
      path = "/data"
      creation_info = {
        owner_uid   = 1000
        owner_gid   = 1000
        permissions = "755"
      }
    }
  }
}

In this case, trying to reach the data under a non-root user with a different uid and gid from the mentioned will fail.

So, first things first, you should ensure the permissions of the AP are set to allow the user to work with the data.

Though, here's the catch. Imagine you've tried the configuration above and understood it doesn't work. After that, you dive into the container image code or the application configuration and learn the user uid:gid working with the drive is 1005:1005. You recreate the AP as there's no way to reconfigure those settings, but it still doesn't work...

This is where it's important to remember that even if you reconfigured the AP settings, they still won't recreate the FS directory nor reconfigure its permissions for you.

access_points = {
  data = {
    posix_user = {
      uid = 1005
      gid = 1005
    }
    root_directory = {
      path = "/data"
      creation_info = {
        owner_uid   = 1005
        owner_gid   = 1005
        permissions = "755"
      }
    }
  }
}

You end up trying to write to the directory owned by the 1000:1000 user under the 1005:1005 user, so it fails. Even if you set permissions to 777 and recreate an AP with it, it won't change anything.

So, you have two options now: either manually adjust the attributes of the directory on FS to fit your new configuration (1005:1005, 755) or create a new directory with the correct settings.