blob: e262098effae4df91e1c2cb6e0e7e44e1b2ded57 [file] [log] [blame]
Request for Comments: New fsal api
This README will be expanded as the work progresses. It will/should
be turned into a fsal writer's HOWTO at some point. This is a work in
progress and therefore still in flux at this point.
State
-----
The server builds and runs to the point of processing exports. Only the
VFS fsal is functional at this point. Debugging continues.
This branch is based on the latest stable tag of the mainline 'next' branch.
The first patch of the new api work is this file, titled:
"Add README.new_api"
Contents
--------
The patch series comes in four parts. The "Revert USE_SHARED configuration
option from code" is the second commit which prepares the code base for the new api.
New infrastructure
These are new files in src/FSAL which implement common
fsal functions.
I have squashed all my incremental changes so that you have
a single file to look at for current state. These are subject
to further change and squashing.
VFS fsal
These are new files that implement the VFS fsal over the
new infrastructure. The main.c file is the fsal object itself
followed by export.c which implements fsal_export for VFS.
handle.c implements fsal_obj_object. File management for handles
is in file.c and extended attributes, not implemented yet for VFS
are in xattr.c.
FSAL initialization
This set of changes to the main server load and initialize
the fsals. Once they are all loaded, fsal_export objects
are created as part of the export list initialization.
Cache Inode changes
Starting with the commit titled "Change cache_inode.h to use new api"
we invade the cache_inode_* code. This series of patches replaces
fsal_handle_t with a pointer to an allocated fsal_obj_handle and
the various FSAL_* calls relevant to it.
The readdir operation has been changed to use a callback. This
greatly simplifies the FSAL method and eliminates multiple loops,
fixed array storage, and a lot of extra work.
File descriptors are removed from the API and will will
be deprecated entirely from cache entries. An fd is a property
of the FSAL and will be managed (cached) there. The VFS fsal
and other POSIX based interfaces have an fd but library based
FSALs have other structures and apis so fileno and friends
are deprecated in the core.
Protocol
These files manage the protocol on one side and access to the
cache entries in CacheInode on the other. The biggest change is
to replace fsal_op_context in the compound structure with
struct user_cred user_credentials. A large number of places where
pcontext was passed around and were (or are no longer) used. These
have simply been removed from the parameter lists.
State and Locking
fsal_op_context pcontext has been for the most part removed from
these files. A few places require user credentials for looking
up handles etc. As with protocol, user_credentials takes its place.
Configuration File Changes
--------------------------
In order to support multiple FSAL definitions in the configuration,
the configuration file now accepts a new syntax for FSAL initialization.
The following fragment defines the loading of the VFS fsal.
###################################################
#
# FSAL parameters.
#
# To use the default value for a parameter,
# just comment the associated line.
#
###################################################
FSAL
{
LogLevel = "Red Alert";
Foo = "baz";
VFS {
FSAL_Shared_Library = "/usr/local/lib/libfsalvfs.so";
# logging level (NIV_FULL_DEBUG, NIV_DEBUG,
# NIV_EVNMT, NIV_CRIT, NIV_MAJ, NIV_NULL)
DebugLevel = "NIV_FULL_DEBUG" ;
# Logging file
#Options: "/var/log/nfs-ganesha.log" or some other file path
# "SYSLOG" prints to syslog
# "STDERR" prints stderr messages to the console that
# started the ganesha process
# "STDOUT" prints stdout messages to the console that
# started the ganesha process
#LogFile = "SYSLOG";
LogFile = "/var/log/nfs-ganesha.log";
# maximum number of simultaneous calls
# to the filesystem.
# ( 0 = no limit ).
max_FS_calls = 0;
}
}
The "FSAL" block has one sub-block, in this example "VFS", for each
fsal it is to load. There can be as many sub-blocks as make sense.
This is the power of trhe new api, to support multiple fsals simultaneously.
There is also provision for FSAL global parameters although none are
actually defined at this point.
The "name" of the FSAL is the tag at the beginning of the sub-block, in this
case "VFS". This name is used for fsal module lookups and diagnostics.
The "FSAL_Shared_Library" value is the absolute path to the module itself.
The path can be anywhere in the local filesystem.
The rest of the parameters are the usual FSAL configuration parameters.
Exports are connected to a specific FSAL in the following fragment:
EXPORT
{
# Export Id (mandatory)
Export_Id = 77 ;
# Exported path (mandatory)
Path = "/home/tmp";
# Exporting FSAL
FSAL = "VFS";
# more export parameters...
}
The "FSAL" keyword defines which FSAL module to use for this export. If
This keyword is missing in an export, the VFS fsal is used. This logic
may change in future to have a server global definition within the
configuration file.
Things to look for
------------------
The new api uses an object oriented design pattern to implement a fsal.
Structure definitions
The fsal_ops_context_t structure is deprecated and all of the
'pcontext' usage has either been replaced or removed. It was
refactored some time back to be common across all FSALs and all
it contained was a pointer to the "export" and user credentials.
* struct compound_data now has 'user_credentials' in place of
'pcontext'. A pointer to these user credentials are passed
through the call stack where access checking applies.
* The use of 'pcontext' in all other cases has been removed.
Nearly all affected function prototypes also had a pointer to
the applicable cache entry 'pentry'. This can be dereferenced
anywhere to get the export, i.e. pentry->handle->export.
The fsal_path_t and fsal_name_t typedefs are not used in the API.
In most instances, they are replaced by a 'const char *'. These
arrays attempt to encapulate string management but they are of
fixed size (mostly too big but also a potential for truncation
or buffer overflows) and both waste space and cause extra structure
copying. Existing core code dereferences the buffer and a NULL
terminated string. They will eventually be deprecated and removed.
The file include/fsal_api.h contains the full fsal api. The
only part visible to the server core is defined here.
Each file that implements an object has its version of the
object definition. One element of this private definition is
the public structure defined in fsal_api.h. The rest of the
elements are private to the fsal itself. These two parts, the
public and private are managed as follows:
* A pointer to the public structure is returned when
the object is created.
* This pointer is used to access methods from it as in:
exp_hdl->ops->lookup(exp_hdl, name, ...);
Note that exp_hdl is used to dereference the method and
it is also *always* the first argument to the
method/function. Think of it as the 'this' argument.
* Inside the method function, the public pointer gets
dereferenced with the following sequence:
struct vfs_fsal_export *myself;
myself = container_of(exp_hdl, struct vfs_fsal_export, export);
The 'container_of' is a macro that takes the public pointer/
handle 'exp_hdl' which is indicated as the element 'export'
of structure type 'vfs_fsal_export'. Throughout the function
where private elements are dereferenced, the 'myself'
pointer is used. 'exp_hdl' is used in the public case.
Object usage
Mutex locks and reference counts are used to manage both concurrent
usage and state. The reference counts are use to determine when the
object is "free". Current use is for managing ref counts and lists.
This will be expanded as more resources such as attributes and open
fds are added.
Since we cannot create objects out of thin air, there is an order
based on one object being the "context" in which the other is
created. In other words, a 'fsal_export' is created from the
'fsal_module' that connects it to the backing store (filesystem).
The same applies to a 'fsal_obj_handle' that only makes sense for
a specific 'fsal_export'.
When an object is created, it is returned with a reference already
taken. The callee of the creating method must then either keep
a persistent reference to it or 'put' it back. For example, a
'fsal_export' gets created for each export in the configuration.
a pointer to it gets saved in exportlist__ and it has a reference
to reflect this. It is now safe to use it to do a 'lookup' which
will return a 'fsal_obj_handle' which can then be kept in a
cache inode entry. If we had done a 'put' on the export, it could
be freed at any point and make a 'lookup' using it unsafe.
In addition to a reference count, object that create other objects
have a list of all the objects they create. This serves two purposes.
The obvious case is to keep the object "busy" until all of its
children are freed. Second, it provides a means to visit all of
the objects it creates.
Every object has a pointer to its parent. This is used for such
things as managing the object list and for calling methods on the
parent.
Pointers vs. Structure copies
In order to manage multiple fsals which have different sized
private object storage, we only pass and save pointers. This keeps
all of the referencing structures of constant size. The
'container_of' manages both the differences in private structure
sizes and the actual layout of the private structures. This also
saves the space and cpu overhead of expensive structure copies.
Public FSAL Structure definitions
The 'ops' element is the most commonly used element of an object's
public structure. Any other elements should be used with care.
Very little upper level code should directly reference
any other elements. Common methods located in
src/FSAL/fsal_commonlib.c are used instead. No size assumptions
should be made either because, other than the 'ops' element, all
other elements are subject to change. Think of them as 'protected'.
The only reason they are declared in the public structure is
because they are used in all fsals i.e. every 'fsal_export' has
lock and reference count.
FSAL structure
--------------
The VFS fsal is the first candidate, primarily because it is the most
promising base for the new work our group is doing. I have followed a
simple pattern for this fsal.
There is one file per object to be implemented. 'export.c' implements
'fsal_export' for VFS and 'main.c' implements 'fsal_module'. This allows
the declaration of the object methods as 'static'. Only common methods
are declared publicly in src/include/FSAL/fsal_commonlib.h and implemented
in src/FSAL/fsal_commonlib.c. Likewise common helper functions are in the
these files.
I have used new names for the public fsal structures on purpose. All of the
old api structures will be deprecated and removed from fsal_types.h and at
that point, the compiler will find the usages that are still 'hiding'
and complain. Otherwise, we could have subtle bugs arise from the change
in structure and especially the use of the 'old' structure.
At one level, it looks like the VFS fsal is a complete rewrite which is
partly true. As with the structures and their names, I am forcing breakage
at compile time.
* There is a significant amount of code re-use. For example, a
'lookup' for a handle is still the same sequence of syscalls or
library calls. That logic and the error paths have all been worked
out.
* Method functions now do only one thing. The 'extra' bit that
acquires attributes, for example, is gone. If the upper layers
want attributes, they should call the method to get them. This
makes things smaller and smaller is good.
* Access checking and all the (see above) is also gone. This is
now moved to the core. The fsal can assume that if the method
is being called, it is ok to do the work. If the action is not
allowed, the core should never make the method call.
* There is a 'test_access' method defined in the api and a common
function is part of the library. Most fsal implementations would
use the library supplied one. However, some implementations may
want to supply their own. There are two very important caveats
with this method:
- This method is only for the core to use. It is *NEVER* called
from within a fsal. Ever.
- If a fsal supplies its own, all that is required is to substitute
the fsal function in the definition of the handle ops vector.
If the fsal implementation would also make this configurable, it
should manage it by doing the test within its own function and
call the library function itself, returning its return values
directly.
* Along with access checking in one place in the core server, the
fsal is responsible for managing the difference between what
NFS wants and the resource (filesystem) it manages provides. For
example, the server assumes that every fsal supports ACLs. If
the resource does not support them or supports them in a different
way, the fsal is responsible for managing the differences.
* There are some linkages between an object and its 'parent' object.
These are managed in two ways. First, common functions are in
fsal_commonlib.c so every fsal implementation can use them. The
second case is for functions that are fsal specific such as
'lookup' which is a 'fsal_export' method that must have intimate
knowledge of what a 'fsal_obj_handle' is. In this case, a
function prototype is defined in export.c so it can be included
in the ops vector and the method is declared global (not static)
in handle.c. Function prototypes are declared in headers only
if they are referenced by multiple modules. The goal is to
break as much incorrect code at the compile level as we can.
Use of _USE_FOO
---------------
There are a number of places where fsal enabling config parameters are used
in the middle of the core. All of this usage will be deprecated and removed.
For example, some data structures have fsal specific elements defined. If the
fsal implementation needs these elements, they should be moved to the
private portion of the most relevant fsal object. This restriction is
necessary to make the core completely fsal implementation agnostic.
There should be no #ifdef conditionals on public data structures. If a feature
is conditionally built, its data structures should be harmless if the feature
is disabled. These conditionals are clutter and can break ABI which is now
important with the new api
NOTE:
There are a few #ifdef conditionals in fsal_api.h. These are temporary
and will be removed in the final version. At that time, the contents
of fsal_api.h will be version controlled as a fixed ABI.