Totally disagree. You have to understand how NFS caching effects your application and what kind of performance and security/availability you need. At that point you can make NFS work for you. You also have to make good decisions about how you implement. This type of use case (long open file waits with concurrent access) are a nightmare with any shared filesystem. Most approaches dealing with possibly stale content are shoulder shrugs.

> This type of use case (long open file waits with concurrent access) are a nightmare with any shared filesystem. Most approaches dealing with possibly stale content are shoulder shrugs.

Well, it is possible to handle correctly, e.g. Lustre. Lustre, however, is very complex compared to NFS, so there's absolutely a price to be paid.

NFS implements close-to-open consistency, which is much weaker than full cache coherency (again, e.g. Lustre).

You want a network shared filesystem then design your data for it.

I think NFS requires to have some knowledge for how you want to use it. The defaults seem reasonable, but I also get confused with all the v3/v4 differences etc. I use it at home to connect everything to my NAS and it works very well for me. My export is this:

  /mnt 192.168.8.0/24(rw,sync,insecure,no_subtree_check,crossmnt,all_squash,anonuid=0,anongid=100)

and my client config with autofs is this:

  rxd01 -fstype=nfs4,ro,soft,noatime,nodiratime,intr,rsize=65536,wsize=65536,nosuid,tcp,allow_other 192.168.8.3:/mnt/rxd01

If you know what each option does, how your network is setup and what your server/clients are capable of, you will eventually find the right settings, but it's not a good OOB experience.

Does anyone know why NFS is such a pain? In the past (10 years ago) I just assumed I was doing it wrong and stopped using it, and have not used it since.

In my experience, the quality of NFS client implementations varies significantly between different operating systems. We made _heavy_ use of NFS for home directories and application backing stores at the University where I used to work, and it was a very good experience -- but this was on Solaris 10 (and later, OpenSolaris) machines. We had heavy NFS client use on many multi-user machines (shell servers, Sun Ray servers, etc) and didn't see reliability problems. On the odd occasion that we needed to reboot the file server for updates, clients would pause and then resume promptly after the server rebooted.

Towards the end of my tenure there, I gave a Linux desktop a try. The NFS experience was amazingly bad by comparison; lots of issues with locking, with becoming disconnected (often until a reboot) from NFS servers, odd performance issues, reliability issues with the automounter, etc.

In the last few months I have tried the NFS client on my current Linux desktop again, thinking things might have improved -- they have, I guess, but not by much. It's still pretty easy for the client to get into a hung state if there's too much packet loss, or if the file server reboots, or whatever. I have to imagine that not enough people are really using Linux NFS clients in anger to drive fixing the issues with it. There is often no escape from the Quality Death Spiral.

NFS requires long admin experience and tuning for each use case. The only comment I can agree with you on is that the linux automounter is lackluster. We used BSD amd for many years with good success.

> NFS requires long admin experience and tuning for each use case.

Depends on what you're going to do with it. For something like sharing home directories, it works well enough.

The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again.

Like the advice to set some specific rsize/wsize settings because the default is too small, oblivious to the fact that the NFS protocol allows the client and server to negotiate maximum sizes, and at least the Linux client and server have taken advantage of this negotiation mechanism for the past 2 decades or so.

Well enough isn't good enough for production. Having worked in high volume, highly available environments for years soho scenarios are not good examples. Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters) or pNFS and gluster require more than tuning mount options. Tuning NFS for a latency averse and throughput intensive application operating on large netcdf and hdf5 file hierarchies is a worthy example.

> Well enough isn't good enough for production.

How informative.

> Having worked in high volume, highly available environments for years soho scenarios are not good examples.

FWIW, I wasn't talking about SOHO. At least in my experience, defaults work well for home & shared work dirs for O(10k) users (not all simultaneously active, though). HA is a pain, though, if you want to DIY, I'll grant you that.

> Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters)

Complex? Sounds like a pretty standard NFS environment.

> or pNFS and gluster require more than tuning mount options.

Yeah, no personal experience there. What did you have to do there?

We did have a clustered NFS appliance for HPC use a decade or so ago. People like to complain how Lustre is a beast to run, but IME Lustre has been smooth sailing compared to the grief that POS gave us. But that wasn't really the fault of the NFS protocol per se, it was just the architecture as well as the implementation of that appliance was crap, particularly so for HPC.

* routine buffer tweak is soho speak. * defaults don't work esp in mixed nfs 3/4 on linux across mixed 1/10 gb segment subnet boundaries. Try and get back to me. You will DOS your file service. * krb5p standard? First I've heard of it. FreeBSD won't do krb5p at nfs4 vanilla via linux nfs server. * Would never do it again. gluster is a shit storm of problems but nice when it works.

It's an extremely old design, designed primarily for read-sharing and with its concurrency features retrofitted. It worked absolutely fine for network-booting diskless Sun workstations in the 90s but its failure modes are just too annoying for modern usage.

Trying to do anything like a database (and the 'git gc' process described is exactly that, a tiny database) over NFS requires the use of very specific techniques to get right.

Sibling commentator has it right - for unreliable WAN networks S3 offers far better semantics, because it's not quite a filesystem.

Posix semantics and networks just dont play well together. Notice that binary storage systems these days avoid the filesystem integration (s3, etc.)

It depends.

Firstly NFS only really works reliably when your network has harmonised UIG/GIDs. Thats the first pain point. This normally means LDAP/AD or shipping /etc/passwd (_shudders_) Also you need to squash root, otherwise people who are local root can do lots of naughty things.

Then you have to make sure that your mountpoint doesn't go away, because stale file handles are a pain in the arse.

Then you have file locking, which causes loads of other pain aswell. Most people turn that off.

After that its mostly alright.

nfsv4 has certain things that are good (pNFS, kerberos, etc) but support was not that great.

It is very good at doing exactly what you don’t want though.