Add hierarchical project quota support

ZFS has a project quota concept, where project is defined with
unique id in dataset, called as project id (aka projid) and files/dirs
inside dataset are assigned to project by setting projid on them.

By associating a project (projid) with specific directory tree in
dataset and setting quota on the projid enforces a directory quota.

When needs to enforce different quota's in directory hierarchy, say
want to limit usage of /d1/d2 to 10G and /d1/d3 also to 10G, this
can be achieved by associating different project with /d1/d2 and /d1/d3
and setting quota on them. Now if want to limit the overall usage of
/d1 to 15G. So need to associate another project with /d1, which should
account the all usage under it. This would enforce a hierarchical
directory quota.

So, to provide hierarchical directory quota, need of supporting multiple
project in same hierarchy, where objects under child project are
accounted in all projects in hierarchy upword. Project quota associated
to directory should be enforced on usage of all of the object inside it
regardless of either it has child quota in hierarchy or not. This means,
if you have projet quota P1 (projid=11) associated to directory /d1 and
child project quota P2 (projid=22) associated to /d1/d2, then usage of
objects (files/dirs/..) under /d1/d2 would be accounted in both
projid=11 and projid=22. Objects specific to /d1 would be accounted in
projid=11.

To account object under child projects in all project in hierarchy up,
each project needs to associated to immediate parent project. To
discover a immediate parent project in hierarchy, project needs to be
associated to root/top inode of project tree.
When associating a project to root/top inode of directory tree, parent
project is found by walking up-word in hierarchy from the inode. Parent
project is found, when it finds a parent inode, on which project id is
set and this project id has inode association. Project to top/root inode
and parent project association is maintained in the new ZAP object on
disk.

Once a new project gets associated with its parent project, also correct
the parent project of all existing projects. There could be a change in
the parent project due to the new project quota. E.g. first associates a
project 11 with /d1, so its parent project is none. Then associates a
project 22 with /d1/d2/d3, so the project of /d1 becomes the parent of
the project associated with /d1/d2/d3. Means, project 11 becomes the
parent project of project 22. And finally associate a project 33 with
/d1/d2, so now 11 becomes the parent project of 33 and 33 becomes the
parent project of 22.

When associating a project to its new parent project (change in parent
project), propagate current usage of project to its new parent project.
Once a hierarchical relationship is established between projects, any
further changes/writes on objects would account usage in objects project
id as well as in its hierarchical parent project ids. To account
existing objects usage under a new project, set project id on directory
recursive. While Setting project id on objects under a project
hierarchy, set the project id on objects if current project id is 0 or
it has a project id where new project id being set is not the
hierarchical parent of current project id on object. If project id being
set is the hierarchical parent of current project id on the object, then
usage of the object is already account-ed in project id being set via
hierarchical association.

E.g. First associated project-id 22 with /d1/d2/d3, So, all objects
inside /d1/d2/d3 are set with project id 22 and their usage gets
account-ed in project 22. Now, when associating a new project 11 with
/d1, project 11 would become a parent project of project 22 and as part
of this association, current usage of project 22 would be propagated to
project 11. After this, setting project id 11 on /d1 recursive would set
project id 11 on all objects until it reaches the /d1/d2/d3 which has a
project id 22 and it would break here as project id 11 being set is
parent project of 22. This effectively would account for the remain-ing
project 11 specific objects usage in it.

In another case, where the project 11 first associated with /d1 and so
usage of all objects under /d1 gets account-ed under project 11. Now,
later associate project 22 with /d1/d2/d3, so the usage of objects
within /d1/d2/d3 would be account-ed in project id 22, while setting
project id 22 on /d1/d2/d3 and underneath objects. As project 11 is
already associated as parent project of 22, so by setting project id 22
on objects from earlier project id 11, it would account the objects
usage in project 22. For project 11, usage would decrement with respect
to changing id from 11 to 22 and increment back with respect to 11 is
hierarchical parent of 22, so effectively no change/delta in usage of
project 11 would happen as expected.

Projects once associated in the hierarchy, its association won’t be
removed until it has an active parent project (quota is set) associated.
Association is needed for hierarchical accounting purposes. Projects
without active parent projects in hierarchy would be dis-associated from
hierarchy. So, when a project is removed (quota is set to none), ZFS
would disassociate non-active projects (quota none), which don't have
parent projects.

Hierarchical quota is enforced by checking project quota on project of
object as well as parent projects in hierarchy.

Signed-off-by: Jitendra Patidar <[email protected]>

Author: jsai20

cmd/zfs/zfs_main.c

@@ -8863,13 +8863,14 @@ zfs_do_project(int argc, char **argv)
 		.zpc_newline = B_TRUE,
 		.zpc_recursive = B_FALSE,
 		.zpc_set_flag = B_FALSE,
+		.zpc_add_hierarchy = B_FALSE,
 	};
 	int ret = 0, c;
 
 	if (argc < 2)
 		usage(B_FALSE);
 
-	while ((c = getopt(argc, argv, "0Ccdkp:rs")) != -1) {
+	while ((c = getopt(argc, argv, "0Ccdkp:rsY")) != -1) {
 		switch (c) {
 		case '0':
 			zpc.zpc_newline = B_FALSE;
@@ -8933,6 +8934,12 @@ zfs_do_project(int argc, char **argv)
 			zpc.zpc_set_flag = B_TRUE;
 			zpc.zpc_op = ZFS_PROJECT_OP_SET;
 			break;
+		case 'Y':
+			zpc.zpc_add_hierarchy = B_TRUE;
+			zpc.zpc_set_flag = B_TRUE;
+			zpc.zpc_recursive = B_TRUE;
+			zpc.zpc_dironly = B_FALSE;
+			break;
 		default:
 			(void) fprintf(stderr, gettext("invalid option '%c'\n"),
 			    optopt);
@@ -8959,13 +8966,25 @@ zfs_do_project(int argc, char **argv)
 			    gettext("'-0' is only valid together with '-c'\n"));
 			usage(B_FALSE);
 		}
+		if (zpc.zpc_add_hierarchy) {
+			(void) fprintf(stderr,
+			    gettext("'-Y' is only valid with set project "
+			    "id\n"));
+			usage(B_FALSE);
+		}
 		break;
 	case ZFS_PROJECT_OP_CHECK:
 		if (zpc.zpc_keep_projid) {
 			(void) fprintf(stderr,
 			    gettext("'-k' is only valid together with '-C'\n"));
 			usage(B_FALSE);
 		}
+		if (zpc.zpc_add_hierarchy) {
+			(void) fprintf(stderr,
+			    gettext("'-Y' is only valid with set project "
+			    "id\n"));
+			usage(B_FALSE);
+		}
 		break;
 	case ZFS_PROJECT_OP_CLEAR:
 		if (zpc.zpc_dironly) {
@@ -8983,6 +9002,12 @@ zfs_do_project(int argc, char **argv)
 			    gettext("'-p' is useless together with '-C'\n"));
 			usage(B_FALSE);
 		}
+		if (zpc.zpc_add_hierarchy) {
+			(void) fprintf(stderr,
+			    gettext("'-Y' is only valid with set project "
+			    "id\n"));
+			usage(B_FALSE);
+		}
 		break;
 	case ZFS_PROJECT_OP_SET:
 		if (zpc.zpc_dironly) {
@@ -9001,6 +9026,13 @@ zfs_do_project(int argc, char **argv)
 			    gettext("'-0' is only valid together with '-c'\n"));
 			usage(B_FALSE);
 		}
+		if (zpc.zpc_add_hierarchy && zpc.zpc_expected_projid ==
+		    ZFS_INVALID_PROJID) {
+			(void) fprintf(stderr,
+			    gettext("'-p' is needed for add hierarchy "
+			    "option '-Y'\n"));
+			usage(B_FALSE);
+		}
 		break;
 	default:
 		ASSERT(0);

cmd/zfs/zfs_project.c

@@ -88,6 +88,11 @@ zfs_project_sanity_check(const char *name, zfs_project_control_t *zpc,
 			    "'-r' option on non-dir target %s\n"), name);
 			return (-1);
 		}
+		if (zpc->zpc_add_hierarchy) {
+			(void) fprintf(stderr, gettext(
+			    "'-Y' option on non-dir target %s\n"), name);
+			return (-1);
+		}
 	}
 
 	return (0);
@@ -118,6 +123,30 @@ zfs_project_load_projid(const char *name, zfs_project_control_t *zpc)
 	return (ret);
 }
 
+static int
+zfs_project_add_hierarchy(const char *name, zfs_project_control_t *zpc)
+{
+	project_hierarchy_arg_t pharg;
+	int ret, fd;
+
+	fd = open(name, O_RDONLY | O_NOCTTY);
+	if (fd < 0) {
+		(void) fprintf(stderr, gettext("failed to open %s: %s\n"),
+		    name, strerror(errno));
+		return (fd);
+	}
+	pharg.pha_projid = zpc->zpc_expected_projid;
+
+	ret = ioctl(fd, FS_IOC_ADD_PROJECT_HIERARCHY, &pharg);
+	if (ret)
+		(void) fprintf(stderr,
+		    gettext("failed to add project hierarchy for %s: %s\n"),
+		    name, strerror(errno));
+
+	close(fd);
+	return (ret);
+}
+
 static int
 zfs_project_handle_one(const char *name, zfs_project_control_t *zpc)
 {
@@ -194,10 +223,12 @@ zfs_project_handle_one(const char *name, zfs_project_control_t *zpc)
 	}
 
 	ret = ioctl(fd, ZFS_IOC_FSSETXATTR, &fsx);
-	if (ret)
+	if (ret && errno != EXDEV) {
 		(void) fprintf(stderr,
 		    gettext("failed to set xattr for %s: %s\n"),
 		    name, strerror(errno));
+	}
+
 
 out:
 	close(fd);
@@ -247,11 +278,13 @@ zfs_project_handle_dir(const char *name, zfs_project_control_t *zpc,
 		ret = zfs_project_handle_one(fullname, zpc);
 		if (!ret && zpc->zpc_recursive && ent->d_type == DT_DIR)
 			zfs_project_item_alloc(head, fullname);
+		if (ret && errno == EXDEV)
+			ret = 0;
 
 		free(fullname);
 	}
 
-	if (errno && !ret) {
+	if (errno && !ret && errno != EXDEV) {
 		ret = -errno;
 		(void) fprintf(stderr, gettext("failed to readdir %s: %s\n"),
 		    name, strerror(errno));
@@ -273,6 +306,11 @@ zfs_project_handle(const char *name, zfs_project_control_t *zpc)
 	if (ret)
 		return (ret);
 
+	if (zpc->zpc_op == ZFS_PROJECT_OP_SET && zpc->zpc_add_hierarchy) {
+		ret = zfs_project_add_hierarchy(name, zpc);
+		if (ret)
+			return (ret);
+	}
 	if ((zpc->zpc_op == ZFS_PROJECT_OP_SET ||
 	    zpc->zpc_op == ZFS_PROJECT_OP_CHECK) &&
 	    zpc->zpc_expected_projid == ZFS_INVALID_PROJID) {
@@ -286,15 +324,20 @@ zfs_project_handle(const char *name, zfs_project_control_t *zpc)
 	if (ret || !S_ISDIR(st.st_mode) || zpc->zpc_dironly ||
 	    (!zpc->zpc_recursive &&
 	    zpc->zpc_op != ZFS_PROJECT_OP_LIST &&
-	    zpc->zpc_op != ZFS_PROJECT_OP_CHECK))
+	    zpc->zpc_op != ZFS_PROJECT_OP_CHECK)) {
+		if (ret && errno == EXDEV)
+			ret = 0;
 		return (ret);
+	}
 
 	list_create(&head, sizeof (zfs_project_item_t),
 	    offsetof(zfs_project_item_t, zpi_list));
 	zfs_project_item_alloc(&head, name);
 	while ((zpi = list_remove_head(&head)) != NULL) {
 		if (!ret)
 			ret = zfs_project_handle_dir(zpi->zpi_name, zpc, &head);
+		if (ret && errno == EXDEV)
+			ret = 0;
 		free(zpi);
 	}
 

cmd/zfs/zfs_projectutil.h

@@ -43,6 +43,7 @@ typedef struct zfs_project_control {
 	boolean_t		zpc_newline;
 	boolean_t		zpc_recursive;
 	boolean_t		zpc_set_flag;
+	boolean_t		zpc_add_hierarchy;
 } zfs_project_control_t;
 
 int zfs_project_handle(const char *name, zfs_project_control_t *zpc);

include/os/freebsd/zfs/sys/zfs_vfsops_os.h

@@ -95,6 +95,7 @@ struct zfsvfs {
 	uint64_t	z_userobjquota_obj;
 	uint64_t	z_groupobjquota_obj;
 	uint64_t	z_projectquota_obj;
+	uint64_t	z_projecthierarchy_obj;
 	uint64_t	z_projectobjquota_obj;
 	uint64_t	z_defaultuserquota;
 	uint64_t	z_defaultgroupquota;

include/os/linux/zfs/sys/zfs_vfsops_os.h

@@ -130,6 +130,7 @@ struct zfsvfs {
 	uint64_t	z_userobjquota_obj;
 	uint64_t	z_groupobjquota_obj;
 	uint64_t	z_projectquota_obj;
+	uint64_t	z_projecthierarchy_obj;
 	uint64_t	z_projectobjquota_obj;
 	uint64_t	z_defaultuserquota;
 	uint64_t	z_defaultgroupquota;

include/os/linux/zfs/sys/zfs_vnops_os.h

@@ -66,6 +66,7 @@ extern int zfs_getattr_fast(zidmap_t *, struct inode *ip, struct kstat *sp);
 #endif
 extern int zfs_setattr(znode_t *zp, vattr_t *vap, int flag, cred_t *cr,
     zidmap_t *mnt_ns);
+extern int zfs_setattr_xattr_dir(znode_t *zp);
 extern int zfs_rename(znode_t *sdzp, char *snm, znode_t *tdzp,
     char *tnm, cred_t *cr, int flags, uint64_t rflags, vattr_t *wo_vap,
     zidmap_t *mnt_ns);

include/sys/dmu_objset.h

@@ -200,6 +200,12 @@ struct objset {
 	dmu_objset_upgrade_cb_t os_upgrade_cb;
 	boolean_t os_upgrade_exit;
 	int os_upgrade_status;
+
+	kmutex_t os_projecthierarchyused_lock;
+	kmutex_t os_projecthierarchyop_lock;
+
+	avl_tree_t os_project_deltas[TXG_SIZE];
+	uint64_t os_projecthierarchy_obj;
 };
 
 #define	DMU_META_OBJSET		0
@@ -268,6 +274,8 @@ int dmu_fsname(const char *snapname, char *buf);
 
 void dmu_objset_evict_done(objset_t *os);
 void dmu_objset_willuse_space(objset_t *os, int64_t space, dmu_tx_t *tx);
+void do_projectusage_update(objset_t *os, uint64_t projid, int64_t used,
+    dmu_tx_t *tx);
 
 void dmu_objset_init(void);
 void dmu_objset_fini(void);

include/sys/fs/zfs.h

@@ -1993,6 +1993,11 @@ enum zio_encrypt {
 	    ZFS_XA_NS_PREFIX_MATCH(LINUX_TRUSTED, name) || \
 	    ZFS_XA_NS_PREFIX_MATCH(LINUX_USER, name))
 
+typedef struct project_hierarchy_arg {
+	uint64_t pha_projid;
+} project_hierarchy_arg_t;
+#define	FS_IOC_ADD_PROJECT_HIERARCHY	_IOW('t', 9, project_hierarchy_arg_t)
+
 #ifdef	__cplusplus
 }
 #endif

include/sys/zfs_quota.h

@@ -38,6 +38,11 @@ extern int zfs_userspace_many(struct zfsvfs *, zfs_userquota_prop_t,
     uint64_t *, void *, uint64_t *, uint64_t *);
 extern int zfs_set_userquota(struct zfsvfs *, zfs_userquota_prop_t,
     const char *, uint64_t, uint64_t);
+extern int zfs_project_hierarchy_add(struct zfsvfs *, uint64_t, uint64_t);
+extern int zfs_project_hierarchy_remove(struct zfsvfs *, uint64_t, uint64_t,
+    boolean_t);
+extern int zfs_projects_are_hierarchical(struct zfsvfs *, uint64_t, uint64_t,
+    boolean_t *);
 
 extern boolean_t zfs_id_overobjquota(struct zfsvfs *, uint64_t, uint64_t);
 extern boolean_t zfs_id_overblockquota(struct zfsvfs *, uint64_t, uint64_t);

include/sys/zfs_znode.h

@@ -140,6 +140,7 @@ extern "C" {
 #define	ZFS_FUID_TABLES		"FUID"
 #define	ZFS_SHARES_DIR		"SHARES"
 #define	ZFS_SA_ATTRS		"SA_ATTRS"
+#define	ZFS_PROJECT_HIERARCHY	"PROJECT_HIERARCHY"
 
 /*
  * Convert mode bits (zp_mode) to BSD-style DT_* values for storing in

module/os/freebsd/zfs/zfs_vfsops.c

@@ -938,6 +938,13 @@ zfsvfs_init(zfsvfs_t *zfsvfs, objset_t *os)
 	else if (error != 0)
 		return (error);
 
+	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_PROJECT_HIERARCHY,
+	    8, 1, &zfsvfs->z_projecthierarchy_obj);
+	if (error == ENOENT)
+		zfsvfs->z_projecthierarchy_obj = 0;
+	else if (error != 0)
+		return (error);
+
 	error = zap_lookup(os, MASTER_NODE_OBJ,
 	    zfs_userquota_prop_prefixes[ZFS_PROP_USEROBJQUOTA],
 	    8, 1, &zfsvfs->z_userobjquota_obj);

module/os/linux/zfs/zfs_vfsops.c

@@ -762,6 +762,14 @@ zfsvfs_init(zfsvfs_t *zfsvfs, objset_t *os)
 	else if (error != 0)
 		return (error);
 
+	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_PROJECT_HIERARCHY,
+	    8, 1, &zfsvfs->z_projecthierarchy_obj);
+	if (error == ENOENT)
+		zfsvfs->z_projecthierarchy_obj = 0;
+	else if (error != 0)
+		return (error);
+	zfsvfs->z_os->os_projecthierarchy_obj = zfsvfs->z_projecthierarchy_obj;
+
 	error = zap_lookup(os, MASTER_NODE_OBJ,
 	    zfs_userquota_prop_prefixes[ZFS_PROP_USEROBJQUOTA],
 	    8, 1, &zfsvfs->z_userobjquota_obj);

module/os/linux/zfs/zfs_vnops_os.c

@@ -2640,6 +2640,50 @@ zfs_setattr(znode_t *zp, vattr_t *vap, int flags, cred_t *cr, zidmap_t *mnt_ns)
 	return (err);
 }
 
+/*
+ * This function sets projid of zp on its xattr dir and each xattr file
+ */
+int
+zfs_setattr_xattr_dir(znode_t *zp)
+{
+	zfsvfs_t *zfsvfs = ZTOZSB(zp);
+	znode_t *xzp = NULL;
+	uint64_t xattr_obj = 0;
+	xoptattr_t *xoap;
+	xvattr_t xva;
+	int err;
+
+	err = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
+	    &xattr_obj, sizeof (xattr_obj));
+	if (err || !xattr_obj)
+		return (err);
+
+	err = zfs_zget(zfsvfs, xattr_obj, &xzp);
+	ASSERT(err == 0);
+	xva_init(&xva);
+	xoap = xva_getxoptattr(&xva);
+	XVA_SET_REQ(&xva, XAT_PROJID);
+	xoap->xoa_projid = zp->z_projid;
+	XVA_SET_REQ(&xva, XAT_PROJINHERIT);
+	xoap->xoa_projinherit = 1;
+
+	cred_t *cr = CRED();
+	crhold(cr);
+	err = zfs_setattr(xzp, (vattr_t *)&xva, 0, cr, zfs_init_idmap);
+	crfree(cr);
+	if (err) {
+		zrele(xzp);
+		return (err);
+	}
+	err = zfs_setattr_dir(xzp);
+	if (err) {
+		zrele(xzp);
+		return (err);
+	}
+	zrele(xzp);
+	return (0);
+}
+
 typedef struct zfs_zlock {
 	krwlock_t	*zl_rwlock;	/* lock we acquired */
 	znode_t		*zl_znode;	/* znode we held */