Hey List.
This is off topic, but I noticed the list has been quiet!!
Trying to get different thoughts on how to approach an issue.
I've got a situation, where I'm going to have a shared nfs, with a bunch of files..
I'm going to have multiple client boxes/(vms) with each client, running apps that access the files on the main nfs.
The idea, is that a clientApp, on a given clientVM will access a file on the nfs, and then the nfs file is no long used. The goal is to rip through/process the nfs files as fast as possible, in a simple manner.
clientVM1 clientApp1
clientVM2 clientApp2
clientVM3 clientApp3 . . clientVMN clientAppN
nfsShare dataFiles...
The process can't have the same data file operated on by multiple clientApps.
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
Surprisingly, in looking around, haven't come up with anything that appears to be really better.
(I've also thought of setting up a "webservice" kind of situation, where clients would get the "file/data" from a webservice, and the webservice, would interface with the nfs dir.. but this would be overkill!)
Any thoughts/comments?
Thanks
On 10/22/2016 06:13 AM, bruce wrote:
The idea, is that a clientApp, on a given clientVM will access a file on the nfs, and then the nfs file is no long used.
It's difficult to offer performance tuning advice without more information about what the app is doing, exactly. Is it reading it into memory and then doing some sort of processing on its in-memory copy?
The process can't have the same data file operated on by multiple clientApps.
Now, that statement makes me think that the application is reading the data, processing it, and then modifying it. If that's the case, then having multiple clients may not be of any benefit. If the applications have to operate in serial, they'll end up doing the work in the same amount of time (or worse) as one system doing all of the work.
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
Probably. You'd have a bunch of clients periodically checking for the lock file before they do work. You'll lose some time to idling when one application finishes, before another application gets the lock and starts working. You may be better off having one process scheduling work, starting up or signaling an application when the data is available. You'll avoid the complexity of the locking, and possibly reduce idle time.
Surprisingly, in looking around, haven't come up with anything that appears to be really better.
Your problem is too specific for a general answer.
On 10/22/2016 12:14 PM, Gordon Messmer wrote:
On 10/22/2016 06:13 AM, bruce wrote:
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
Probably. You'd have a bunch of clients periodically checking for the lock file before they do work. You'll lose some time to idling when one application finishes, before another application gets the lock and starts working. You may be better off having one process scheduling work, starting up or signaling an application when the data is available. You'll avoid the complexity of the locking, and possibly reduce idle time.
So, you are suggesting: One process on the nfs server that knows about all the files in question and waits for clients to ask for a file (any file????), returns back to that nfs client an nfs handle for some specific file chosen by the server, creates some flag somewhere (on the server) that the file has been served to username xyz, uid cde, on client a.b.c. When is finished with that file, does something like ask the server to delete that file, or says to the server I am done with that file; at which point server might delete the file per client request, or simply remove the flag that the file is in use by username xyz, uid cde, on client a.b.c, ... and so on. Well, now you have another wait problem for each request for some unserved file. In such a serial service of requests - clients might time out, and can err out or simply retry. What if the list of files is in the thousands? Serially serving said files would be very time consuming.
Parallel server processes: The server process might be coded so that for each request, it forks a child process to serve the request. Now, child processes have to compete for a lock on the list of files to be served, one of the children will succeed, and set the flag (mark username xyz, uid cde, on client a.b.c has file /filename/), serve the file, unlock the list and exit. Same client that requested the file must then request the server to delete the file or tell the server it is done with the file - again requiring the child server process (not same child process that served the file, but a new child spawned to handle the request) to lock the list + delete the file or simply remove the flag of the client.
On the surface of it, it sounds like this is a slow process. But it is not.
A sever process that spawns children that compete for the list is a good way to serve said files.
On Sat, Oct 22, 2016 at 4:00 PM, jd1008 jd1008@gmail.com wrote:
On 10/22/2016 12:14 PM, Gordon Messmer wrote:
On 10/22/2016 06:13 AM, bruce wrote:
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
Probably. You'd have a bunch of clients periodically checking for the lock file before they do work. You'll lose some time to idling when one application finishes, before another application gets the lock and starts working. You may be better off having one process scheduling work, starting up or signaling an application when the data is available. You'll avoid the complexity of the locking, and possibly reduce idle time.
So, you are suggesting: One process on the nfs server that knows about all the files in question and waits for clients to ask for a file (any file????), returns back to that nfs client an nfs handle for some specific file chosen by the server, creates some flag somewhere (on the server) that the file has been served to username xyz, uid cde, on client a.b.c. When is finished with that file, does something like ask the server to delete that file, or says to the server I am done with that file; at which point server might delete the file per client request, or simply remove the flag that the file is in use by username xyz, uid cde, on client a.b.c, ... and so on. Well, now you have another wait problem for each request for some unserved file. In such a serial service of requests - clients might time out, and can err out or simply retry. What if the list of files is in the thousands? Serially serving said files would be very time consuming.
Parallel server processes: The server process might be coded so that for each request, it forks a child process to serve the request. Now, child processes have to compete for a lock on the list of files to be served, one of the children will succeed, and set the flag (mark username xyz, uid cde, on client a.b.c has file /filename/), serve the file, unlock the list and exit. Same client that requested the file must then request the server to delete the file or tell the server it is done with the file - again requiring the child server process (not same child process that served the file, but a new child spawned to handle the request) to lock the list + delete the file or simply remove the flag of the client.
On the surface of it, it sounds like this is a slow process. But it is not.
A sever process that spawns children that compete for the list is a good way to serve said files.
----- Hey JD!
Thanks for the reply.
I thought about having some sort of "nfs/server" side process that iterates through the list of files, and then determines when a "client" is ready, and pushes the file to the client.
In having the clients, pull the file (set the pidFILE), there's the possibility that the pidFile, could be messed up, along with a few other minor issues, but it would be simple to implement.
Thanks
-b
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
On 10/22/2016 02:39 PM, bruce wrote:
On Sat, Oct 22, 2016 at 4:00 PM, jd1008 jd1008@gmail.com wrote:
On 10/22/2016 12:14 PM, Gordon Messmer wrote:
On 10/22/2016 06:13 AM, bruce wrote:
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
Probably. You'd have a bunch of clients periodically checking for the lock file before they do work. You'll lose some time to idling when one application finishes, before another application gets the lock and starts working. You may be better off having one process scheduling work, starting up or signaling an application when the data is available. You'll avoid the complexity of the locking, and possibly reduce idle time.
So, you are suggesting: One process on the nfs server that knows about all the files in question and waits for clients to ask for a file (any file????), returns back to that nfs client an nfs handle for some specific file chosen by the server, creates some flag somewhere (on the server) that the file has been served to username xyz, uid cde, on client a.b.c. When is finished with that file, does something like ask the server to delete that file, or says to the server I am done with that file; at which point server might delete the file per client request, or simply remove the flag that the file is in use by username xyz, uid cde, on client a.b.c, ... and so on. Well, now you have another wait problem for each request for some unserved file. In such a serial service of requests - clients might time out, and can err out or simply retry. What if the list of files is in the thousands? Serially serving said files would be very time consuming.
Parallel server processes: The server process might be coded so that for each request, it forks a child process to serve the request. Now, child processes have to compete for a lock on the list of files to be served, one of the children will succeed, and set the flag (mark username xyz, uid cde, on client a.b.c has file /filename/), serve the file, unlock the list and exit. Same client that requested the file must then request the server to delete the file or tell the server it is done with the file - again requiring the child server process (not same child process that served the file, but a new child spawned to handle the request) to lock the list + delete the file or simply remove the flag of the client.
On the surface of it, it sounds like this is a slow process. But it is not.
A sever process that spawns children that compete for the list is a good way to serve said files.
Hey JD!
Thanks for the reply.
I thought about having some sort of "nfs/server" side process that iterates through the list of files, and then determines when a "client" is ready, and pushes the file to the client.
In having the clients, pull the file (set the pidFILE), there's the possibility that the pidFile, could be messed up, along with a few other minor issues, but it would be simple to implement.
Thanks
-b
Hi Bruce, Client's .pid files are not shared. They only exist on the client. So, client A would inevitable end up processing the same file as client B.
I suggest you write a small daemon on the nfs server that does what I suggest. The "list" is simply an array of strings (array of pointers to null terminated character strings.). This array should be in an include file that also exists on the client that will build the client application code. Once built it can be installed on all clients. the server process will also include this file. The server process should 1. setup the socket for listening to requests. 2. loop listen for incoming requests accept the request (which contains data of the nature of the request) spawn a thread and pass to it the acceptance socket (the data of which the thread will process). endloop
3. The thread will parse the request to see if it is a request for a file or a request to close a file or a request to delete a file. The thread will use sychronization primitives to lock the list mentioned above and perform the request. If the request is for a file, then it needs to open that file for reading, and return an nfs handle to the client, do some overhead, like create the flag of client info (username, uid, client ip), then unlock the list and return. The parent process will not wait for the child thread, as it must simply accept connections and spawn the same thread to handle the request. Multiply (I do not mean mathematical 'multiply', I mean more than one), spawned threads are independent of each other. They only share the the list and the list's lock primitives.
The client will use that handle as a file descriptor and process the file. When client is done, client will request to a: close the file or b: close and delete the file.
That's it.
On 10/22/2016 03:13 PM, bruce wrote:
I've thought of having a "pid" file on the nfs, where each clientApp would reset/lock/use the pidFile to then get/use/delete the file as required, but this seems tb slow.. but doable.
I would experiment with using the atomicity of the filesystem rename.
The process on client number N will do:
1) list files in directory 2) identify a filename not starting with "INPROGRESS-" (e.g. "abcd"); if file-not-found {sleep(something); goto 1} 3) rename "abcd" to "INPROGRESS-N-abcd"; if file-not-found goto 2 4) process INPROGRESS-N-abcd 5) delete INPROGRESS-N-abcd 6) goto 1
The rename is atomic, so if two processes try to grab the same file, only one wins the race; when a file has been renamed, nobody else can mess with it.
The good part is that you do not need anything running on the server.
Of course you have some tunables; sleep could be 0.01s or 1m, depending on your expectations; "INPROGRESS-" could be a smarter Unicode prefix, for example "⚒".
(alternatively, keep the name unchanged but rename the file away to a client-specific directory)
Just be sure that rename maintains the atomicity for the NFS implementation and version you are using.
Regards.
On Sat, Oct 22, 2016 at 1:00 PM, jd1008 jd1008@gmail.com wrote:
Parallel server processes: The server process might be coded so that for each request, it forks a child process to serve the request. Now, child processes have to compete for a lock on the list of files to be served, one of the children will succeed
I honestly can't think of any good reason to spawn multiple children that compete for a lock to do work, when the process that's starting them *should* have adequate information about which clients are active and which are idle, and can use that information to select a client on which to start the process.
I'd also point out that this is a well-solved problem. There are quite a few very good parallelization schedulers already available (particularly for HTC purposes), so there's no need to write one for this purpose.
On 10/24/2016 01:28 PM, Gordon Messmer wrote:
I honestly can't think of any good reason to spawn multiple children that compete for a lock to do work, when the process that's starting them*should* have adequate information about which clients are active and which are idle, and can use that information to select a client on which to start the process.
A database running on a server probably doesn't want more than one process to be modifying it at any given time.
On 10/24/2016 02:07 PM, Joe Zeff wrote:
A database running on a server probably doesn't want more than one process to be modifying it at any given time.
Yes, I get that. But why would it be better for the scheduler to start up database servers on multiple hosts and allow them to compete for a lock, rather than pick one host to start the database and start one instance? The former is what JD is suggesting (adapted to your proposal; as far as I can tell), while I suggest that the latter is probably a better approach if it's possible, and it usually is.
On 10/24/2016 02:34 PM, Gordon Messmer wrote:
Yes, I get that. But why would it be better for the scheduler to start up database servers on multiple hosts and allow them to compete for a lock, rather than pick one host to start the database and start one instance? The former is what JD is suggesting (adapted to your proposal; as far as I can tell), while I suggest that the latter is probably a better approach if it's possible, and it usually is.
I don't have the experience to judge which one is better; I was only pointing out a reason that you'd not want multiple instances of the database program working on the same file at any given time.
On 10/24/2016 03:16 PM, Joe Zeff wrote:
On 10/24/2016 02:34 PM, Gordon Messmer wrote:
Yes, I get that. But why would it be better for the scheduler to start up database servers on multiple hosts and allow them to compete for a lock, rather than pick one host to start the database and start one instance? The former is what JD is suggesting (adapted to your proposal; as far as I can tell), while I suggest that the latter is probably a better approach if it's possible, and it usually is.
I don't have the experience to judge which one is better; I was only pointing out a reason that you'd not want multiple instances of the database program working on the same file at any given time.
A suggestion for this...use some scheduler like RabbitMQ. The database puts process requests up on the RabbitMQ queue, the worker machines pull requests off the queue and processes them. That way, only one machine processes one file at a time and you don't have any race conditions, PID file contentions, etc.. You can spin up additional workers as necessary to parallelize your operations to whatever level you want.
Yes, it takes some coding, but I suspect that was expected. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - On a scale of 1 to 10 I'd say... oh, somewhere in there. - ----------------------------------------------------------------------
On 10/24/2016 03:16 PM, Joe Zeff wrote:
I don't have the experience to judge which one is better; I was only pointing out a reason that you'd not want multiple instances of the database program working on the same file at any given time.
Yes, but I wasn't proposing starting multiple instances. I was suggesting the opposite: that instead of starting multiple instances and using a lock file to allow just one to work at a time, a scheduler should start just one client.
On 10/26/2016 03:33 PM, Gordon Messmer wrote:
On 10/24/2016 03:16 PM, Joe Zeff wrote:
I don't have the experience to judge which one is better; I was only pointing out a reason that you'd not want multiple instances of the database program working on the same file at any given time.
Yes, but I wasn't proposing starting multiple instances. I was suggesting the opposite: that instead of starting multiple instances and using a lock file to allow just one to work at a time, a scheduler should start just one client.
Yes, I understood that you were trying to avoid having more than one instance working on the file at any given time. I was trying to explain to a different list member why that's often a Good Thing.
-
bruce -
Gordon Messmer -
JD -
Joe Zeff -
Rick Stevens -
Roberto Ragusa