Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an enhancement, or whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level paths:
/mnt/foo /mnt/foo/bar
each defined by their own map objects. SSSD does not handle this (neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP query in a sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so many clients and a large list, this might be better left to the client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
Thanks again
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an enhancement, or whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level paths:
/mnt/foo /mnt/foo/bar
each defined by their own map objects. SSSD does not handle this (neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP query in a sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so many clients and a large list, this might be better left to the client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks again _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
In persistent mounts (found in /etc/fstab), there's two settings that tell the init process in which phase to run fsck and in which phase to mount. So in persistent mounts, it's easy to order the mounts during boot time. If necessary. And (as you say, the "_netdev" flag tells the boot process to mount a fs very late.)
I'm not sure that sorting the lines in /etc/fstab would accomplish this, I thought modern Linux versions fsck in parallel now, not sequentially. (I know they probe multiple SCSI buses in parallel.)
But I suppose autofs has no concept of mount phase?
Spike
On Wed, Dec 4, 2019 at 4:36 AM Pavel Březina pbrezina@redhat.com wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in
my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an enhancement, or
whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level paths:
/mnt/foo /mnt/foo/bar
each defined by their own map objects. SSSD does not handle this
(neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only
get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and
get around this issue.
Would it be possible to have SSSD return the maps from LDAP query in a
sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, (
https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so
many clients and a large list, this might be better left to the client to do instead.
I'm happy to help out if someone can point me in the right direction in
the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks again _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
Can someone explain what mount phase is, along with an example?
On 12/4/19 9:26 AM, Spike White wrote:
In persistent mounts (found in /etc/fstab), there's two settings that tell the init process in which phase to run fsck and in which phase to mount. So in persistent mounts, it's easy to order the mounts during boot time. If necessary. And (as you say, the "_netdev" flag tells the boot process to mount a fs very late.)
I'm not sure that sorting the lines in /etc/fstab would accomplish this, I thought modern Linux versions fsck in parallel now, not sequentially. (I know they probe multiple SCSI buses in parallel.)
But I suppose autofs has no concept of mount phase?
Spike
On Wed, Dec 4, 2019 at 4:36 AM Pavel Březina <pbrezina@redhat.com mailto:pbrezina@redhat.com> wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote: > Hi everyone. > > First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there. > > I am not sure if this is the right way to ask for an enhancement, or whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required) > > Basically for an automount map where I need nested top level paths: > > /mnt/foo > /mnt/foo/bar > > each defined by their own map objects. SSSD does not handle this (neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as: > > /mnt/foo/bar > /mnt/foo > > the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only get the entries from /mnt/foo > > Using file based mount maps, one can easily sort the top level maps, and get around this issue. > Would it be possible to have SSSD return the maps from LDAP query in a sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, ( > https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so many clients and a large list, this might be better left to the client to do instead. > > I'm happy to help out if someone can point me in the right direction in the code. SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself. CCing Ian, do you have any thoughts on this? > > Thanks again > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org <mailto:sssd-users@lists.fedorahosted.org> > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org <mailto:sssd-users-leave@lists.fedorahosted.org> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org <mailto:sssd-users@lists.fedorahosted.org> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org <mailto:sssd-users-leave@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o...
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an enhancement, or whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level paths:
/mnt/foo /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is not needed.
Nesting of mounts can only be done by using mount map entries with offsets.
each defined by their own map objects. SSSD does not handle this (neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP query in a sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so many clients and a large list, this might be better left to the client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Such a map might not make a lot of sense if you were designing from scratch, but what I'm actually doing is mapping our Windows DFS namespace "\foo" in Active Directory where a lot of apps have been written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly minimize the effort when having these apps migrate and run under linux, or in some situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux (mostly python apps)
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything works fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the maps perfectly fine in automap -m but the order in which they're returned results in some of the parent paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod, followed by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any traversal to a path under /foo/x meant that ALL mounts got mounted at once, which was quite undesirable.
Would you have a recommendation on how this might be better authored in ldap? As I said, I must unfortunately follow the namespace as is.
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an enhancement, or whether I should file an issue on GitHub, but I am running into an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level paths:
/mnt/foo /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is not needed.
Nesting of mounts can only be done by using mount map entries with offsets.
each defined by their own map objects. SSSD does not handle this (neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and you'll only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP query in a sorted way? There is an LDAP control that most LDAP servers support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but with so many clients and a large list, this might be better left to the client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Ok, thanks for that.
Such a map might not make a lot of sense if you were designing from scratch, but what I'm actually doing is mapping our Windows DFS namespace "\foo" in Active Directory where a lot of apps have been written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly minimize the effort when having these apps migrate and run under linux, or in some situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux (mostly python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything works fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the maps perfectly fine in automap -m but the order in which they're returned results in some of the parent paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod, followed by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any traversal to a path under /foo/x meant that ALL mounts got mounted at once, which was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on what distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the config it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts which could prevent (at least some of the sensitivity) to the mount on scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better authored in ldap? As I said, I must unfortunately follow the namespace as is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's in use and the mount ends up not behaving properly. Then, after the kernel mounts list gets disjointed they demand to know why this happened and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't even work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm pretty sure that could break other functionality people might have come to expect to behave a certain way.
Consequently I would like to find another way to achieve this if we can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases) have a utility to set basic mount junctions. Have you investigated this, autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage your mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an
enhancement,
or whether I should file an issue on GitHub, but I am running
into
an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level
paths:
/mnt/foo /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is not needed.
Nesting of mounts can only be done by using mount map entries with offsets.
each defined by their own map objects. SSSD does not handle
this
(neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and
you'll
only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP
query
in a sorted way? There is an LDAP control that most LDAP
servers
support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but
with
so many clients and a large list, this might be better left to
the
client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
Thank you for thinking about this. I will investigate the submounts you suggested as well as the NFS junctions.
I am on the latest centos/rhel 7.x line
On Fri, Dec 6, 2019, 2:39 AM Ian Kent ikent@redhat.com wrote:
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Ok, thanks for that.
Such a map might not make a lot of sense if you were designing from scratch, but what I'm actually doing is mapping our Windows DFS namespace "\foo" in Active Directory where a lot of apps have been written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly minimize the effort when having these apps migrate and run under linux, or in some situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux (mostly python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything works fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the maps perfectly fine in automap -m but the order in which they're returned results in some of the parent paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod, followed by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any traversal to a path under /foo/x meant that ALL mounts got mounted at once, which was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on what distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the config it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts which could prevent (at least some of the sensitivity) to the mount on scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better authored in ldap? As I said, I must unfortunately follow the namespace as is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's in use and the mount ends up not behaving properly. Then, after the kernel mounts list gets disjointed they demand to know why this happened and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't even work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm pretty sure that could break other functionality people might have come to expect to behave a certain way.
Consequently I would like to find another way to achieve this if we can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases) have a utility to set basic mount junctions. Have you investigated this, autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage your mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD. It's easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an
enhancement,
or whether I should file an issue on GitHub, but I am running
into
an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login required)
Basically for an automount map where I need nested top level
paths:
/mnt/foo /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is not needed.
Nesting of mounts can only be done by using mount map entries with offsets.
each defined by their own map objects. SSSD does not handle
this
(neither does LDAP directly from autofs) because the return map from LDAP is unsorted, and if the maps are provided to autofs ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and
you'll
only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top level maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP
query
in a sorted way? There is an LDAP control that most LDAP
servers
support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control ) but
with
so many clients and a large list, this might be better left to
the
client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do not think it should be done in SSSD (unless autofs interface says so) but rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
On Fri, 2019-12-06 at 07:34 -0500, Oguzhan Eris wrote:
Thank you for thinking about this. I will investigate the submounts you suggested as well as the NFS junctions.
I am on the latest centos/rhel 7.x line
So you should have "browse_mode = no" in your installed configuration unless you're nuked the default installed config and created a new one from scratch without that option in a provisioning system or you retain and old /etc/sysconfig/autofs that overrides it.
So, supposing this isn't the case, unless you add a "browse" option to mounts it should be disabled so that mount point directories aren't pre-created for indirect mount map keys.
In that case find(1) won't see the mount point directories and won't be able to try and access them.
On Fri, Dec 6, 2019, 2:39 AM Ian Kent ikent@redhat.com wrote:
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Following on from above this should be able to be done by using autofs submounts so that offsets aren't used at all and together with not enabling mount point browsing this should go a long way to resolving the mount storm problem you see.
Aside: stat family system calls aren't meant to trigger automounts (really only applies to indirect mounts, direct mounts and offset mounts always trigger mounts because of the way they behave) but way back when this was done the statfs(2) system call was missed and continued to trigger mounts. And IIRC find(1) uses statfs() during tree traversal and we get mount storms. Now it's too late to change that but there's also a case that statfs() should trigger mounts (rather than requiring a trailing "/" as with making other stat family calls trigger mounts) since a statfs() of an autofs fs path is pretty much useless ...
Now I haven't tested this at all but from the above this is what I think might help.
In the master map (auto.master or your master map fragment) use: /foo /etc/automap/auto.foo
where auto.foo is: x --fstype=autofs /etc/automap/auto.x z --fstype=autofs /etc/automap/auto.z
and auto.x is: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2 dev --fstype=autofs /etc/automap/auto.x_dev prod --fstype=autofs /etc/automap/auto.x_prod
and auto.x_dev is: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
and auto.x_prod is: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
and finally auto.z is: dir5 -fstype=nfs4 server5:/share_dir5
It's not as compact and simple as using offsets allone but it should prevent the mount storm.
Converting this to be stored in LDAP should be straight forward too since you've already had to do that for some of the maps.
Ian
Ok, thanks for that.
Such a map might not make a lot of sense if you were designing
from
scratch, but what I'm actually doing is mapping our Windows DFS namespace "\foo" in Active Directory where a lot of apps have
been
written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly minimize
the
effort when having these apps migrate and run under linux, or in
some
situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux
(mostly
python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything
works
fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the maps perfectly fine in automap -m but the order in which they're returned results in some of the
parent
paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod,
followed
by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any traversal to
a
path under /foo/x meant that ALL mounts got mounted at once,
which
was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on what distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the config it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts which could prevent (at least some of the sensitivity) to the mount on scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better
authored
in ldap? As I said, I must unfortunately follow the namespace
as
is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's in use and the mount ends up not behaving properly. Then, after the kernel mounts list gets disjointed they demand to know why this happened and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't even work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm pretty sure that could break other functionality people might have come to expect to behave a certain way.
Consequently I would like to find another way to achieve this if we can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases) have a utility to set basic mount junctions. Have you investigated this, autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage your mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote:
Hi everyone.
First off, thanks to everyone who's ever worked on SSSD.
It's
easily in my top 5 favorite FOSS projects out there.
I am not sure if this is the right way to ask for an
enhancement,
or whether I should file an issue on GitHub, but I am
running
into
an issue that's described in this Red Hat page https://access.redhat.com/solutions/3673501 (login
required)
Basically for an automount map where I need nested top
level
paths:
/mnt/foo /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is
not
needed.
Nesting of mounts can only be done by using mount map entries with offsets.
each defined by their own map objects. SSSD does not
handle
this
(neither does LDAP directly from autofs) because the return
map
from LDAP is unsorted, and if the maps are provided to
autofs
ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
/mnt/foo/bar /mnt/foo
the /mnt/foo map masks the previous /mnt/foo/bar map and
you'll
only get the entries from /mnt/foo
Using file based mount maps, one can easily sort the top
level
maps, and get around this issue. Would it be possible to have SSSD return the maps from LDAP
query
in a sorted way? There is an LDAP control that most LDAP
servers
support to return a sorted output, ( https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control )
but
with
so many clients and a large list, this might be better left
to
the
client to do instead.
I'm happy to help out if someone can point me in the right direction in the code.
SSSD is just a data provider, if some sorting is needed I do
not
think it should be done in SSSD (unless autofs interface says so)
but
rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
Hi Ian
I must apologize as I forgot to mention that I do in fact use browse mode. For the windows->Linux migration, it helps because we are moving to a case sensitive world, and people need to know the exact case of the mount points.
I will still try out submounts as you suggested to see if the mount storm is any better than the hierarchial mounts.
Thanks once again.
On Sat, Dec 7, 2019, 11:44 PM Ian Kent ikent@redhat.com wrote:
On Fri, 2019-12-06 at 07:34 -0500, Oguzhan Eris wrote:
Thank you for thinking about this. I will investigate the submounts you suggested as well as the NFS junctions.
I am on the latest centos/rhel 7.x line
So you should have "browse_mode = no" in your installed configuration unless you're nuked the default installed config and created a new one from scratch without that option in a provisioning system or you retain and old /etc/sysconfig/autofs that overrides it.
So, supposing this isn't the case, unless you add a "browse" option to mounts it should be disabled so that mount point directories aren't pre-created for indirect mount map keys.
In that case find(1) won't see the mount point directories and won't be able to try and access them.
On Fri, Dec 6, 2019, 2:39 AM Ian Kent ikent@redhat.com wrote:
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Following on from above this should be able to be done by using autofs submounts so that offsets aren't used at all and together with not enabling mount point browsing this should go a long way to resolving the mount storm problem you see.
Aside: stat family system calls aren't meant to trigger automounts (really only applies to indirect mounts, direct mounts and offset mounts always trigger mounts because of the way they behave) but way back when this was done the statfs(2) system call was missed and continued to trigger mounts. And IIRC find(1) uses statfs() during tree traversal and we get mount storms. Now it's too late to change that but there's also a case that statfs() should trigger mounts (rather than requiring a trailing "/" as with making other stat family calls trigger mounts) since a statfs() of an autofs fs path is pretty much useless ...
Now I haven't tested this at all but from the above this is what I think might help.
In the master map (auto.master or your master map fragment) use: /foo /etc/automap/auto.foo
where auto.foo is: x --fstype=autofs /etc/automap/auto.x z --fstype=autofs /etc/automap/auto.z
and auto.x is: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2 dev --fstype=autofs /etc/automap/auto.x_dev prod --fstype=autofs /etc/automap/auto.x_prod
and auto.x_dev is: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
and auto.x_prod is: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
and finally auto.z is: dir5 -fstype=nfs4 server5:/share_dir5
It's not as compact and simple as using offsets allone but it should prevent the mount storm.
Converting this to be stored in LDAP should be straight forward too since you've already had to do that for some of the maps.
Ian
Ok, thanks for that.
Such a map might not make a lot of sense if you were designing
from
scratch, but what I'm actually doing is mapping our Windows DFS namespace "\foo" in Active Directory where a lot of apps have
been
written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly minimize
the
effort when having these apps migrate and run under linux, or in
some
situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux
(mostly
python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything
works
fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the maps perfectly fine in automap -m but the order in which they're returned results in some of the
parent
paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod,
followed
by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any traversal to
a
path under /foo/x meant that ALL mounts got mounted at once,
which
was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on what distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the config it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts which could prevent (at least some of the sensitivity) to the mount on scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better
authored
in ldap? As I said, I must unfortunately follow the namespace
as
is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's in use and the mount ends up not behaving properly. Then, after the kernel mounts list gets disjointed they demand to know why this happened and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't even work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm pretty sure that could break other functionality people might have come to expect to behave a certain way.
Consequently I would like to find another way to achieve this if we can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases) have a utility to set basic mount junctions. Have you investigated this, autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage your mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote:
On 11/30/19 5:41 PM, Oguzhan Eris wrote: > Hi everyone. > > First off, thanks to everyone who's ever worked on SSSD.
It's
> easily in my top 5 favorite FOSS projects out there. > > I am not sure if this is the right way to ask for an
enhancement,
> or whether I should file an issue on GitHub, but I am
running
into
> an issue that's described in this Red Hat page > https://access.redhat.com/solutions/3673501 (login
required)
> > Basically for an automount map where I need nested top
level
paths:
> > /mnt/foo > /mnt/foo/bar
Those look like master map entries or perhaps direct mount map entries.
Which is it?
In either case nesting mounts is not supported so sorting is
not
needed.
Nesting of mounts can only be done by using mount map entries with offsets.
> > each defined by their own map objects. SSSD does not
handle
this
> (neither does LDAP directly from autofs) because the return
map
> from LDAP is unsorted, and if the maps are provided to
autofs
> ordered as:
So each has their own map, but you haven't provided a complete master map entry or map examples so I still can't tell what you actually want to do.
I'll assume these are master map entries ... and the maps are called auto.foo and auto.bar, and they are indirect mount maps, and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough, due to the fact that mount dependency changes can (and do) cause problems that are extremely hard to resolve, nesting won't be supported in any other way.
> > /mnt/foo/bar > /mnt/foo > > the /mnt/foo map masks the previous /mnt/foo/bar map and
you'll
> only get the entries from /mnt/foo > > Using file based mount maps, one can easily sort the top
level
> maps, and get around this issue. > Would it be possible to have SSSD return the maps from LDAP
query
> in a sorted way? There is an LDAP control that most LDAP
servers
> support to return a sorted output, ( > https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control )
but
with
> so many clients and a large list, this might be better left
to
the
> client to do instead. > > I'm happy to help out if someone can point me in the right > direction in the code.
SSSD is just a data provider, if some sorting is needed I do
not
think it should be done in SSSD (unless autofs interface says so)
but
rather in autofs itself.
CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use case might still be needed, we'll see.
Ian
On Sun, 2019-12-08 at 10:04 -0500, Oguzhan Eris wrote:
Hi Ian
I must apologize as I forgot to mention that I do in fact use browse mode. For the windows->Linux migration, it helps because we are moving to a case sensitive world, and people need to know the exact case of the mount points.
Right but with it disabled in the configuration you have the option to add or remove the "browse" option in your master maps as you need.
Personally I think leaving the global setting disabled and using the option as needed is a better choice.
Unfortunately, if the mount point directories exist find will see them an poke its nose into them but OTOH seeing those directories is what you expect from a normal file system ... cant win ...
I will still try out submounts as you suggested to see if the mount storm is any better than the hierarchial mounts.
Thanks once again.
My pleasure.
On Sat, Dec 7, 2019, 11:44 PM Ian Kent ikent@redhat.com wrote:
On Fri, 2019-12-06 at 07:34 -0500, Oguzhan Eris wrote:
Thank you for thinking about this. I will investigate the
submounts
you suggested as well as the NFS junctions.
I am on the latest centos/rhel 7.x line
So you should have "browse_mode = no" in your installed configuration unless you're nuked the default installed config and created a new one from scratch without that option in a provisioning system or you retain and old /etc/sysconfig/autofs that overrides it.
So, supposing this isn't the case, unless you add a "browse" option to mounts it should be disabled so that mount point directories aren't pre-created for indirect mount map keys.
In that case find(1) won't see the mount point directories and won't be able to try and access them.
On Fri, Dec 6, 2019, 2:39 AM Ian Kent ikent@redhat.com wrote:
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Following on from above this should be able to be done by using autofs submounts so that offsets aren't used at all and together with not enabling mount point browsing this should go a long way to resolving the mount storm problem you see.
Aside: stat family system calls aren't meant to trigger automounts (really only applies to indirect mounts, direct mounts and offset mounts always trigger mounts because of the way they behave) but way back when this was done the statfs(2) system call was missed and continued to trigger mounts. And IIRC find(1) uses statfs() during tree traversal and we get mount storms. Now it's too late to change that but there's also a case that statfs() should trigger mounts (rather than requiring a trailing "/" as with making other stat family calls trigger mounts) since a statfs() of an autofs fs path is pretty much useless ...
Now I haven't tested this at all but from the above this is what I think might help.
In the master map (auto.master or your master map fragment) use: /foo /etc/automap/auto.foo
where auto.foo is: x --fstype=autofs /etc/automap/auto.x z --fstype=autofs /etc/automap/auto.z
and auto.x is: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2 dev --fstype=autofs /etc/automap/auto.x_dev prod --fstype=autofs /etc/automap/auto.x_prod
and auto.x_dev is: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
and auto.x_prod is: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
and finally auto.z is: dir5 -fstype=nfs4 server5:/share_dir5
It's not as compact and simple as using offsets allone but it should prevent the mount storm.
Converting this to be stored in LDAP should be straight forward too since you've already had to do that for some of the maps.
Ian
Ok, thanks for that.
Such a map might not make a lot of sense if you were
designing
from
scratch, but what I'm actually doing is mapping our Windows
DFS
namespace "\foo" in Active Directory where a lot of apps
have
been
written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly
minimize
the
effort when having these apps migrate and run under linux, or
in
some
situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux
(mostly
python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit
differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything
works
fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the
maps
perfectly fine in automap -m but the order in which they're returned results in some of
the
parent
paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod,
followed
by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the
start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any
traversal to
a
path under /foo/x meant that ALL mounts got mounted at once,
which
was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on
what
distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the
config
it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts
which
could prevent (at least some of the sensitivity) to the mount
on
scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better
authored
in ldap? As I said, I must unfortunately follow the
namespace
as
is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for
it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's
in
use and the mount ends up not behaving properly. Then, after the
kernel
mounts list gets disjointed they demand to know why this
happened
and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't
even
work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm
pretty
sure that could break other functionality people might have
come to
expect to behave a certain way.
Consequently I would like to find another way to achieve this
if we
can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases)
have
a utility to set basic mount junctions. Have you investigated
this,
autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage
your
mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com
wrote:
On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote: > On 11/30/19 5:41 PM, Oguzhan Eris wrote: > > Hi everyone. > > > > First off, thanks to everyone who's ever worked on
SSSD.
It's
> > easily in my top 5 favorite FOSS projects out there. > > > > I am not sure if this is the right way to ask for an enhancement, > > or whether I should file an issue on GitHub, but I am
running
into > > an issue that's described in this Red Hat page > > https://access.redhat.com/solutions/3673501 (login
required)
> > > > Basically for an automount map where I need nested top
level
paths: > > > > /mnt/foo > > /mnt/foo/bar
Those look like master map entries or perhaps direct mount
map
entries.
Which is it?
In either case nesting mounts is not supported so sorting
is
not
needed.
Nesting of mounts can only be done by using mount map
entries
with offsets.
> > > > each defined by their own map objects. SSSD does not
handle
this > > (neither does LDAP directly from autofs) because the
return
map
> > from LDAP is unsorted, and if the maps are provided to
autofs
> > ordered as:
So each has their own map, but you haven't provided a
complete
master map entry or map examples so I still can't tell what
you
actually want to do.
I'll assume these are master map entries ... and the maps
are
called auto.foo and auto.bar, and they are indirect mount
maps,
and I'll use file maps for the example.
In the master map: /mnt /etc/auto.foo
In /etc/auto.foo:
foo / foo.server.net:/path/to/foo /bar -type=autofs /etc/auto.bar
might be something like what you need to do.
Allowing nesting in offset mounts this problematic enough,
due
to the fact that mount dependency changes can (and do)
cause
problems that are extremely hard to resolve, nesting won't
be
supported in any other way.
> > > > /mnt/foo/bar > > /mnt/foo > > > > the /mnt/foo map masks the previous /mnt/foo/bar map
and
you'll > > only get the entries from /mnt/foo > > > > Using file based mount maps, one can easily sort the
top
level
> > maps, and get around this issue. > > Would it be possible to have SSSD return the maps from
LDAP
query > > in a sorted way? There is an LDAP control that most
LDAP
servers > > support to return a sorted output, ( > >
https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control )
but
with > > so many clients and a large list, this might be better
left
to
the > > client to do instead. > > > > I'm happy to help out if someone can point me in the
right
> > direction in the code. > > SSSD is just a data provider, if some sorting is needed I
do
not
> think > it should be done in SSSD (unless autofs interface says
so)
but
> rather > in autofs itself. > > CCing Ian, do you have any thoughts on this?
Thanks Pavel, some more specific information about the use
case
might still be needed, we'll see.
Ian
On Mon, 2019-12-09 at 07:50 +0800, Ian Kent wrote:
On Sun, 2019-12-08 at 10:04 -0500, Oguzhan Eris wrote:
Hi Ian
I must apologize as I forgot to mention that I do in fact use browse mode. For the windows->Linux migration, it helps because we are moving to a case sensitive world, and people need to know the exact case of the mount points.
Right but with it disabled in the configuration you have the option to add or remove the "browse" option in your master maps as you need.
Personally I think leaving the global setting disabled and using the option as needed is a better choice.
Unfortunately, if the mount point directories exist find will see them an poke its nose into them but OTOH seeing those directories is what you expect from a normal file system ... cant win ...
One last thing comes to mind.
find(1) has the -xautofs option to prevent descending into autofs file systems. If the mount storms are due to scripts maintained by yourselves and you only need to avoid traversals of autofs file systems completely you could use this.
I will still try out submounts as you suggested to see if the mount storm is any better than the hierarchial mounts.
Thanks once again.
My pleasure.
On Sat, Dec 7, 2019, 11:44 PM Ian Kent ikent@redhat.com wrote:
On Fri, 2019-12-06 at 07:34 -0500, Oguzhan Eris wrote:
Thank you for thinking about this. I will investigate the
submounts
you suggested as well as the NFS junctions.
I am on the latest centos/rhel 7.x line
So you should have "browse_mode = no" in your installed configuration unless you're nuked the default installed config and created a new one from scratch without that option in a provisioning system or you retain and old /etc/sysconfig/autofs that overrides it.
So, supposing this isn't the case, unless you add a "browse" option to mounts it should be disabled so that mount point directories aren't pre-created for indirect mount map keys.
In that case find(1) won't see the mount point directories and won't be able to try and access them.
On Fri, Dec 6, 2019, 2:39 AM Ian Kent ikent@redhat.com wrote:
On Thu, 2019-12-05 at 23:09 -0500, Oguzhan Eris wrote:
Thanks for the reply. My file based maps are as such: basically blank auto.master that includes auto.master.d /etc/auto.master.d/foo.autofs: /foo/x /etc/automap/x.txt /foo/x/dev /etc/automap/x_dev.txt foo/x/prod /etc/automap/x_prod.txt /foo/z /etc/automap/z.txt
/etc/automap/x.txt: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2
/etc/automap/x_dev.txt: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
/etc/automap/x_prod.txt: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
/etc/automap/z.txt: dir5 -fstype=nfs4 server5:/share_dir5
Following on from above this should be able to be done by using autofs submounts so that offsets aren't used at all and together with not enabling mount point browsing this should go a long way to resolving the mount storm problem you see.
Aside: stat family system calls aren't meant to trigger automounts (really only applies to indirect mounts, direct mounts and offset mounts always trigger mounts because of the way they behave) but way back when this was done the statfs(2) system call was missed and continued to trigger mounts. And IIRC find(1) uses statfs() during tree traversal and we get mount storms. Now it's too late to change that but there's also a case that statfs() should trigger mounts (rather than requiring a trailing "/" as with making other stat family calls trigger mounts) since a statfs() of an autofs fs path is pretty much useless ...
Now I haven't tested this at all but from the above this is what I think might help.
In the master map (auto.master or your master map fragment) use: /foo /etc/automap/auto.foo
where auto.foo is: x --fstype=autofs /etc/automap/auto.x z --fstype=autofs /etc/automap/auto.z
and auto.x is: dir1 -fstype=nfs4 server1:/share_dir1 dir2 -fstype=nfs4 server2:/share_dir2 dev --fstype=autofs /etc/automap/auto.x_dev prod --fstype=autofs /etc/automap/auto.x_prod
and auto.x_dev is: dir3 -fstype=nfs4 devserver1:/share_dir3 dir4 -fstype=nfs4 devserver2:/share_dir4
and auto.x_prod is: dir3 -fstype=nfs4 prodserver1:/share_dir3 dir4 -fstype=nfs4 prodserver2:/share_dir4
and finally auto.z is: dir5 -fstype=nfs4 server5:/share_dir5
It's not as compact and simple as using offsets allone but it should prevent the mount storm.
Converting this to be stored in LDAP should be straight forward too since you've already had to do that for some of the maps.
Ian
Ok, thanks for that.
Such a map might not make a lot of sense if you were
designing
from
scratch, but what I'm actually doing is mapping our Windows
DFS
namespace "\foo" in Active Directory where a lot of apps
have
been
written to use just one global namespace, to now be available under linux in the same way at /foo . This will greatly
minimize
the
effort when having these apps migrate and run under linux, or
in
some
situations, be completely cross platform apps with very minor settings to them to make them work under windows vs. linux
(mostly
python apps)
LOL, the why people do what they do is not really my place to question. The fact you feel you need to do it is enough for me. But, for various reasons, it may need to be done a bit
differently.
Anyway, so as I have it above, as long as my indirect map in foo.autofs is ordered the way I have above, then everything
works
fine, and I will end up with:
/foo/x /foo/x/dir1 /foo/x/dir2 /foo/x/dev/ /foo/x/dev/dir3 /foo/x/dev/dir4 /foo/x/prod/ /foo/x/prod/dir3 /foo/x/prod/dir4 /foo/z
When I model this in LDAP (via sssd) however, I can see the
maps
perfectly fine in automap -m but the order in which they're returned results in some of
the
parent
paths masking their nested children. i.e. if /foo/x/dev gets mounted first, and then /foo/x/prod,
followed
by /foo/x It will result in me seeing just /foo/x/dir1 /foo/x/dir2
The moment I umount /foo/x I will then see /foo/x/dev/... and /foo/x/prod/...
Hopefully that makes sense.
Yes, I got the idea of the problem you were having at the
start.
I had previously done hierarchical maps with something like:
auto.master /foo /etc/automount/foo.txt
/etc/automount/foo.txt: x \ dir1 ... \ dir2 .... \ dev/dir3 ....\ dev/dir4 ... \ prod/dir3 ...\ prod/dir4 ...
And that worked, but it had the huge burden that any
traversal to
a
path under /foo/x meant that ALL mounts got mounted at once,
which
was quite undesirable.
Yes, that can happen with this type of map entry.
This sort of construct probably could be done using submounts similar to the example I gave previously.
I guess I should ask what version of autofs your using and on
what
distribution?
Do you set browse_mode in your autofs configuration?
If browse_mode is set to "no" (if it's not present in the
config
it's default is yes) then mount point directories in indirect mounts won't be seen until mounted so they won't get mounted when the tree is scanned ... but with that construct you have above those offsets will behave like direct mounts which will be present and will respond to scans.
It may be possible to break this tree down to use submounts
which
could prevent (at least some of the sensitivity) to the mount
on
scan problem you have.
I'll need to think about it.
Would you have a recommendation on how this might be better
authored
in ldap? As I said, I must unfortunately follow the
namespace
as
is.
LDAP, and NIS and NISPLUS all have no ordering guarantee.
Strictly speaking I don't allow nesting in anything other than those hierarchical mount map entries but I don't explicitly check for
it.
And I don't really want to put that sort of a limitation into autofs.
But the problem happens all to often (with hierarchical map entries) that someone rips out an export in the midst of a tree that's
in
use and the mount ends up not behaving properly. Then, after the
kernel
mounts list gets disjointed they demand to know why this
happened
and I can't give an answer because something like "you did something you should have known not to do" doesn't go down all that well.
Inevitably I can't reproduce these types of problem so I can't
even
work out if I can improve things ...
Also, while I'm tempted to just sort the master map list I'm
pretty
sure that could break other functionality people might have
come to
expect to behave a certain way.
Consequently I would like to find another way to achieve this
if we
can. I'll give that some thought.
Your mounting NFS mounts, could you use the NFS cross-mount automounting here?
I'm pretty sure nfs-utils (but perhaps fairly recent releases)
have
a utility to set basic mount junctions. Have you investigated
this,
autofs should play well with the NFS automounting?
I guess that takes away the benefit of being able to manage
your
mounts in one place using one method ...
I'll have a bit of a think about this one. Ian
Thanks again
On Wed, Dec 4, 2019 at 7:20 PM Ian Kent ikent@redhat.com
wrote:
> On Wed, 2019-12-04 at 11:36 +0100, Pavel Březina wrote: > > On 11/30/19 5:41 PM, Oguzhan Eris wrote: > > > Hi everyone. > > > > > > First off, thanks to everyone who's ever worked on
SSSD.
It's
> > > easily in my top 5 favorite FOSS projects out there. > > > > > > I am not sure if this is the right way to ask for an > enhancement, > > > or whether I should file an issue on GitHub, but I > > > am
running
> into > > > an issue that's described in this Red Hat page > > > https://access.redhat.com/solutions/3673501 (login
required)
> > > Basically for an automount map where I need nested > > > top
level
> paths: > > > /mnt/foo > > > /mnt/foo/bar > > Those look like master map entries or perhaps direct > mount
map
> entries. > > Which is it? > > In either case nesting mounts is not supported so sorting
is
not
> needed. > > Nesting of mounts can only be done by using mount map
entries
> with offsets. > > > > each defined by their own map objects. SSSD does not
handle
> this > > > (neither does LDAP directly from autofs) because the
return
map
> > > from LDAP is unsorted, and if the maps are provided > > > to
autofs
> > > ordered as: > > So each has their own map, but you haven't provided a
complete
> master map entry or map examples so I still can't tell > what
you
> actually want to do. > > I'll assume these are master map entries ... and the maps
are
> called auto.foo and auto.bar, and they are indirect mount
maps,
> and I'll use file maps for the example. > > In the master map: > /mnt /etc/auto.foo > > In /etc/auto.foo: > > foo / foo.server.net:/path/to/foo > /bar -type=autofs /etc/auto.bar > > might be something like what you need to do. > > Allowing nesting in offset mounts this problematic > enough,
due
> to the fact that mount dependency changes can (and do)
cause
> problems that are extremely hard to resolve, nesting > won't
be
> supported in any other way. > > > > /mnt/foo/bar > > > /mnt/foo > > > > > > the /mnt/foo map masks the previous /mnt/foo/bar map
and
> you'll > > > only get the entries from /mnt/foo > > > > > > Using file based mount maps, one can easily sort the
top
level
> > > maps, and get around this issue. > > > Would it be possible to have SSSD return the maps > > > from
LDAP
> query > > > in a sorted way? There is an LDAP control that most
LDAP
> servers > > > support to return a sorted output, ( > > >
https://ldapwiki.com/wiki/Server%20Side%20Sort%20Control )
but
> with > > > so many clients and a large list, this might be > > > better
left
to
> the > > > client to do instead. > > > > > > I'm happy to help out if someone can point me in the
right
> > > direction in the code. > > > > SSSD is just a data provider, if some sorting is needed > > I
do
not
> > think > > it should be done in SSSD (unless autofs interface says
so)
but
> > rather > > in autofs itself. > > > > CCing Ian, do you have any thoughts on this? > > Thanks Pavel, some more specific information about the > use
case
> might still be needed, we'll see. > > Ian >
sssd-users@lists.fedorahosted.org
-
Goetz, Patrick G -
Ian Kent -
Oguzhan Eris -
Pavel Březina -
Spike White