4:25 a.m.
Greetings, friends of all things DevAssistant.
For eleven minor versions, DevAssistant has been the tool to set up
your environment on your host system, on bare metal. That works very
well, but an increasing number of developers can't use it for their
use case. Why, you ask? Containers. While DevAssistant has very basic
support for containers in the 'Docker' assistant, it still does all
the magic on the host. What we need is a full-fledged containerised
app support, and DevAssistant will have to change quite fundamentally
to offer it.
Right now, DevAssistant is pretty much a monolithic application that
runs on the host machine, requires GTK+ for its GUI and makes some
assumptions about what the environment looks like. That makes it quite
hard to make it work reliably under Mac OS X or Windows, heck even
different Linux distros. To overcome this problem, I suggest
splitting DevAssistant based on the client/server architecture.
The basic idea is that the server runs in a privileged container or on
the host system, and very thin client connects to it and exchange
information about assistants and commands through an API (for example
using JSON). You may think about it as "DevAssistant as a service".
When developing this architecture, I have heavily borrowed from
Cockpit, the web-based sysadmin tool developed for Fedora, which uses
a webserver to accept commands, and then, through a bridge app, issues
calls to various system services. DevAssistant then should be
reorganised into these components:
The client
----------
While DevAssistant uses already a sort of a client-server
architecture, the clients are still quite thick, and not easily
replaceable. Our intention is to put only the bare minimum into the
client - an engine to construct a user interface from the received
data. That means that the command line interface will construct a help
message with assistants and commands based on what is available on the
server side. Similarly, the WebUI will construct a form based on
fields that the server sends to it, generated from the arguments
specified in an assistant. The client always runs under the user that
wishes to run an assistant.
I am acutely aware of the fact that by using the web interface for
running commands under the root user, I am effectively delegating the
security to the browser, but I doubt we currently have the resources
to have a new full-fledged GUI based on some desktop graphical
toolkit.
The webserver
-------------
The webserver is the front-facing part of the entire server, which
interacts with the client. What it does is spin up a "bridge" instance
(see below) for the user under whom the "client" is running, and pass
messages between the two. The webserver itself doesn't touch the
system or containers in any way, and will probably run under a unique
user, for example 'devassistant'. I expect it will take the list of
available users from /etc/passwd. I still don't know what webserver to
use, but security is definitely what we should look at first.
The bridge
----------
This is the core part of the new system, which will contain most of
today's DevAssistant package. This part runs under the user who
actually wants to execute the assistants, and is spawned by the
webserver. It loads the available assistants, runs them with
parameters supplied through the webserver, and if necessary, delegates
the execution of superuser tasks to the "service" with elevated
privileges.
The service
-----------
Since the bridge runs under a user that doesn't necessarily have all
the permissions an assistant may require, there will be a service
(DBus?) that accepts commands and executes them as the super user.
When such a command is invoked, the service will ask an authentication
agent if the user is authorised to perform such a command, and based
on the agent's reply will carry it out appropriately. If an additional
authentication is required, the bridge sends a request to the client
for proper authentication.
The agent
---------
The last part of the system is an authentication agent. This agent,
based on its policies, decides if a given user is permitted to carry
out a command passed to the "service", and return a True/False reply.
This will probably be a PolicyKit service.
This is the general plan of remodelling DevAssistant to better suit
containerised development. I would like to know your ideas about or
criticisms of this outline, as it may shape the development for months
or years to come.
Cheers, Tomas Radej
6:56 a.m.
Thanks for sending this, Tomas. My comments are inline (since we've discussed this
together previously, I'm ok with this, I just have some minor nitpicks :)).
----- Original Message -----
Greetings, friends of all things DevAssistant.
For eleven minor versions, DevAssistant has been the tool to set up
your environment on your host system, on bare metal. That works very
well, but an increasing number of developers can't use it for their
use case. Why, you ask? Containers. While DevAssistant has very basic
support for containers in the 'Docker' assistant, it still does all
the magic on the host. What we need is a full-fledged containerised
app support, and DevAssistant will have to change quite fundamentally
to offer it.
... while trying to not discard the current use-cases, where possible. My perception of
this is that we should pursue this goal as much as we can. If we need to sacrifice some
generality of DA along the way, then so be it - but only if it's absolutely
necessary.
Right now, DevAssistant is pretty much a monolithic application that
runs on the host machine, requires GTK+ for its GUI and makes some
assumptions about what the environment looks like. That makes it quite
hard to make it work reliably under Mac OS X or Windows, heck even
different Linux distros. To overcome this problem, I suggest
splitting DevAssistant based on the client/server architecture.
The basic idea is that the server runs in a privileged container or on
the host system,
Or in a virtual machine, or possibly even on a remote machine.
and very thin client connects to it and exchange
information about assistants and commands through an API (for example
using JSON). You may think about it as "DevAssistant as a service".
When developing this architecture, I have heavily borrowed from
Cockpit, the web-based sysadmin tool developed for Fedora, which uses
a webserver to accept commands, and then, through a bridge app, issues
calls to various system services. DevAssistant then should be
reorganised into these components:
The client
----------
While DevAssistant uses already a sort of a client-server
architecture, the clients are still quite thick, and not easily
replaceable. Our intention is to put only the bare minimum into the
client - an engine to construct a user interface from the received
data. That means that the command line interface will construct a help
message with assistants and commands based on what is available on the
server side. Similarly, the WebUI will construct a form based on
fields that the server sends to it, generated from the arguments
specified in an assistant. The client always runs under the user that
wishes to run an assistant.
I am acutely aware of the fact that by using the web interface for
running commands under the root user, I am effectively delegating the
security to the browser, but I doubt we currently have the resources
to have a new full-fledged GUI based on some desktop graphical
toolkit.
I even think that doing our own GUI would be vulnerable to a hundred and one different CVS
that we wouldn't know about. Using web browser is both convenient, easy to implement
and doesn't require any special software installed on user's machine.
The webserver
-------------
The webserver is the front-facing part of the entire server, which
interacts with the client. What it does is spin up a "bridge" instance
(see below) for the user under whom the "client" is running, and pass
messages between the two. The webserver itself doesn't touch the
system or containers in any way, and will probably run under a unique
user, for example 'devassistant'. I expect it will take the list of
available users from /etc/passwd. I still don't know what webserver to
use, but security is definitely what we should look at first.
IMO flask is secure enough, while also being lightweight and flexible. I don't have a
strong opinion here, so feel free to propose a different solution.
<snip>
This is the general plan of remodelling DevAssistant to better suit
containerised development. I would like to know your ideas about or
criticisms of this outline, as it may shape the development for months
or years to come.
I think that there's one side effect that this will have that you didn't point out
- it will be pretty much impossible to install DA by "pip install" in this
paradigm. It'd be close to impossible to do manage enabling system services, creating
"devassistant" user etc. with "pip install". I'm ok with this,
since that's the way I've wanted to move anyway. I just think it's worth
pointing out :)
Again, thanks a lot, Tomas!
Slavek
Cheers, Tomas Radej
8:33 a.m.
On 28.4.2015 13:56, Bohuslav Kabrda wrote:
IMO flask is secure enough, while also being lightweight and
flexible. I don't have a strong opinion here, so feel free to propose a different
solution.
I'm not saying this is better than Flask, just a tip to look at: Bottle
http://bottlepy.org/
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
8:30 a.m.
On 27.4.2015 11:25, Tomas Radej wrote:
The basic idea is that the server runs in a privileged container or
on
the host system, and very thin client connects to it and exchange
information about assistants and commands through an API (for example
using JSON). You may think about it as "DevAssistant as a service".
Here I'm not sure what do you refer as a server and basically, what runs
where. I'll try to make some assumptions bellow, please correct me if I
get it wrong.
The client
----------
While DevAssistant uses already a sort of a client-server
architecture, the clients are still quite thick, and not easily
replaceable. Our intention is to put only the bare minimum into the
client - an engine to construct a user interface from the received
data. That means that the command line interface will construct a help
message with assistants and commands based on what is available on the
server side. Similarly, the WebUI will construct a form based on
fields that the server sends to it, generated from the arguments
specified in an assistant. The client always runs under the user that
wishes to run an assistant.
So we'll have multiple clients: CLI and WebUI. Both running on the
system the developer is primly using to interact with DA as a process
owned by the developer, e.g. "joe".
The webserver
-------------
The webserver is the front-facing part of the entire server, which
interacts with the client. What it does is spin up a "bridge" instance
(see below) for the user under whom the "client" is running, and pass
messages between the two. The webserver itself doesn't touch the
system or containers in any way, and will probably run under a unique
user, for example 'devassistant'. I expect it will take the list of
available users from /etc/passwd. I still don't know what webserver to
use, but security is definitely what we should look at first.
Now, where runs this? I would expect this to be run in a "privileged
container" as mentioned above. Why do we care under what user this runs
than? I'm afraid this is a bit confusing to me.
The bridge
----------
This is the core part of the new system, which will contain most of
today's DevAssistant package. This part runs under the user who
actually wants to execute the assistants, and is spawned by the
webserver. It loads the available assistants, runs them with
parameters supplied through the webserver, and if necessary, delegates
the execution of superuser tasks to the "service" with elevated
privileges.
I'm again confused by the user thing. This is supposed to by run as
"joe" again? Spawned by the webserver that does not run under "joe",
but
under "devassistant" user. Is that even possible, if "devassistant"
user
has no root ACLs?
Or is this also run in a container? Different one or the same one as
webserver?
The service
-----------
Since the bridge runs under a user that doesn't necessarily have all
the permissions an assistant may require, there will be a service
(DBus?) that accepts commands and executes them as the super user.
When such a command is invoked, the service will ask an authentication
agent if the user is authorised to perform such a command, and based
on the agent's reply will carry it out appropriately. If an additional
authentication is required, the bridge sends a request to the client
for proper authentication.
How does this works on non-Linux platforms than?
The agent
---------
The last part of the system is an authentication agent. This agent,
based on its policies, decides if a given user is permitted to carry
out a command passed to the "service", and return a True/False reply.
This will probably be a PolicyKit service.
The same question.
Thanks.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:46 a.m.
Hi,
On 29/04/15 15:30, Miro Hrončok wrote:
On 27.4.2015 11:25, Tomas Radej wrote:
> The basic idea is that the server runs in a privileged container or on
> the host system, and very thin client connects to it and exchange
> information about assistants and commands through an API (for example
> using JSON). You may think about it as "DevAssistant as a service".
Here I'm not sure what do you refer as a server and basically, what runs
where. I'll try to make some assumptions bellow, please correct me if I
get it wrong.
The "server" in client/server is everything described below that is not
"The client".
> The client
> ----------
>
> While DevAssistant uses already a sort of a client-server
> architecture, the clients are still quite thick, and not easily
> replaceable. Our intention is to put only the bare minimum into the
> client - an engine to construct a user interface from the received
> data. That means that the command line interface will construct a help
> message with assistants and commands based on what is available on the
> server side. Similarly, the WebUI will construct a form based on
> fields that the server sends to it, generated from the arguments
> specified in an assistant. The client always runs under the user that
> wishes to run an assistant.
So we'll have multiple clients: CLI and WebUI. Both running on the
system the developer is primly using to interact with DA as a process
owned by the developer, e.g. "joe".
The CLI client will run under "joe" (in Linux, Mac OS, Windows if
applicable). The WebUI runs in the browser by typing IP:Port and logging in.
The webserver runs under "devassistant" (a new user with special,
constrained set of permissions), or possibly under "joe" as well when
installed from pip. In the latter case, some fallbacks will be necessary
(see below).
> The webserver
> -------------
>
> The webserver is the front-facing part of the entire server, which
> interacts with the client. What it does is spin up a "bridge" instance
> (see below) for the user under whom the "client" is running, and pass
> messages between the two. The webserver itself doesn't touch the
> system or containers in any way, and will probably run under a unique
> user, for example 'devassistant'. I expect it will take the list of
> available users from /etc/passwd. I still don't know what webserver to
> use, but security is definitely what we should look at first.
Now, where runs this? I would expect this to be run in a "privileged
container" as mentioned above. Why do we care under what user this runs
than? I'm afraid this is a bit confusing to me.
Privileged container or bare metal (if packaged in the distro or
installed via pip). DA could then have some fallback solution that would
enable running DA from pip or source via sudo or the like, in order to
keep compatibility. I am not sure if the benefits of doing that would
outweigh the work invested, though.
Ad why we care: It's the "bare metal" case. I'd imagine having only one
instance of the webserver run for all users on the machine is beneficial
(avoids collisions, can run on a predictable default port, i. e. no need
to specify connection details when running the client).
> The bridge
> ----------
>
> This is the core part of the new system, which will contain most of
> today's DevAssistant package. This part runs under the user who
> actually wants to execute the assistants, and is spawned by the
> webserver. It loads the available assistants, runs them with
> parameters supplied through the webserver, and if necessary, delegates
> the execution of superuser tasks to the "service" with elevated
> privileges.
I'm again confused by the user thing. This is supposed to by run as
"joe" again? Spawned by the webserver that does not run under "joe",
but
under "devassistant" user. Is that even possible, if "devassistant"
user
has no root ACLs?
I think it should be possible to give the "devassistant" user limited
root access solely for spawning the "bridge" instances under different
users via sudoers, polkit policy or the like. A trivial solution that
comes to my mind is having a root-only shell script that takes a
username and spawns the 'bridge' instance.
On the other hand, if you run DA from pip, it could be possible to run
everything under the regular user ("joe"). We sort of assume the person
who runs DevAssistant has root access anyway.
Or is this also run in a container? Different one or the same one as
webserver?
It is, in the default containerised scenario. I would argue for putting
all the server bits into the same (privileged) container, if possible.
> The service
> -----------
>
> Since the bridge runs under a user that doesn't necessarily have all
> the permissions an assistant may require, there will be a service
> (DBus?) that accepts commands and executes them as the super user.
> When such a command is invoked, the service will ask an authentication
> agent if the user is authorised to perform such a command, and based
> on the agent's reply will carry it out appropriately. If an additional
> authentication is required, the bridge sends a request to the client
> for proper authentication.
How does this works on non-Linux platforms than?
DA server (i. e. webserver, bridge, service, agent) runs in a virtual
machine or in a container. It is my impression that supporting OS X is
quite hard with the current architecture, and Windows is next to impossible.
By moving DevAssistant to a more constrained, virtualised environment,
it will be much easier to support more complex use cases - e. g.
inheritable container support for *all* "create" projects. Furthermore,
the only part running on the host system will be the client. With WebUI,
supporting OS X or Windows would be dead easy, because all the changes
would be done inside the VM.
As for how would DA be distributed then, I am thinking a pre-made
virtual image of sorts with DA fully set-up. In any case, we can't help
Windows users set up virtualisation, so they'll have to do some work
anyway. If we decide to go this way, they'll only need to install a
virtualisation host, run the image, and connect to IP:Port from their
browser.
This sort of limits the usage on non-Linux platforms, but I believe that
encouraging the use of virtualised environments is something we should do.
> The agent
> ---------
>
> The last part of the system is an authentication agent. This agent,
> based on its policies, decides if a given user is permitted to carry
> out a command passed to the "service", and return a True/False reply.
> This will probably be a PolicyKit service.
The same question.
The same answer. When run on bare metal (from pip, under "joe"), DA
could fall back to running sudo from the "bridge" as it does now.
Thanks.
TR
4:17 a.m.
On 30.4.2015 10:46, Tomas Radej wrote:
Hi,
On 29/04/15 15:30, Miro Hrončok wrote:
> On 27.4.2015 11:25, Tomas Radej wrote:
>> The basic idea is that the server runs in a privileged container or on
>> the host system, and very thin client connects to it and exchange
>> information about assistants and commands through an API (for example
>> using JSON). You may think about it as "DevAssistant as a service".
>
> Here I'm not sure what do you refer as a server and basically, what runs
> where. I'll try to make some assumptions bellow, please correct me if I
> get it wrong.
The "server" in client/server is everything described below that is not
"The client".
...
Thanks Tomáš for explaining everything, I think I understand it much
better now.
> How does this works on non-Linux platforms than?
DA server (i. e. webserver, bridge, service, agent) runs in a virtual
machine or in a container. It is my impression that supporting OS X is
quite hard with the current architecture, and Windows is next to
impossible.
By moving DevAssistant to a more constrained, virtualised environment,
it will be much easier to support more complex use cases - e. g.
inheritable container support for *all* "create" projects. Furthermore,
the only part running on the host system will be the client. With WebUI,
supporting OS X or Windows would be dead easy, because all the changes
would be done inside the VM.
As for how would DA be distributed then, I am thinking a pre-made
virtual image of sorts with DA fully set-up. In any case, we can't help
Windows users set up virtualisation, so they'll have to do some work
anyway. If we decide to go this way, they'll only need to install a
virtualisation host, run the image, and connect to IP:Port from their
browser.
This sort of limits the usage on non-Linux platforms, but I believe that
encouraging the use of virtualised environments is something we should do.
One thing I don't understand, when I run DA on Windows/Mac int this way,
and I create let's say a new Django project using the WebUI, it's
created in the VM. How do I develop something in this project than? I
hope not using some text editor inside the VM, because frankly, if I do
that I could have just easily install Fedora into a VM and run DA as it
is today.
Do I get my source code mounted? Or something else?
How do I run ./manage.py runserver and such?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7:50 a.m.
On 30/04/15 11:17, Miro Hrončok wrote:
Thanks Tomáš for explaining everything, I think I understand it much
better now.
Any time.
>> How does this works on non-Linux platforms than?
>
> DA server (i. e. webserver, bridge, service, agent) runs in a virtual
> machine or in a container. It is my impression that supporting OS X is
> quite hard with the current architecture, and Windows is next to
> impossible.
>
> By moving DevAssistant to a more constrained, virtualised environment,
> it will be much easier to support more complex use cases - e. g.
> inheritable container support for *all* "create" projects. Furthermore,
> the only part running on the host system will be the client. With WebUI,
> supporting OS X or Windows would be dead easy, because all the changes
> would be done inside the VM.
>
> As for how would DA be distributed then, I am thinking a pre-made
> virtual image of sorts with DA fully set-up. In any case, we can't help
> Windows users set up virtualisation, so they'll have to do some work
> anyway. If we decide to go this way, they'll only need to install a
> virtualisation host, run the image, and connect to IP:Port from their
> browser.
>
> This sort of limits the usage on non-Linux platforms, but I believe that
> encouraging the use of virtualised environments is something we should do.
One thing I don't understand, when I run DA on Windows/Mac int this way,
and I create let's say a new Django project using the WebUI, it's
created in the VM. How do I develop something in this project than? I
hope not using some text editor inside the VM, because frankly, if I do
that I could have just easily install Fedora into a VM and run DA as it
is today.
Do I get my source code mounted? Or something else?
Mounting directories is the plan, yes. I assume a considerable part of
getting DA running in such a setup will have to be solved through
documentation, but it is my estimate that it's still significantly less
work for the user than if done any other way.
How do I run ./manage.py runserver and such?
Good question. One way that crossed my mind is having generic assistants
that just wrap said invocation. Another is making sure sshd is running
and just telling the user to SSH into the VM/container.
The third, currently rather hypothetical way, is automatically
generating custom assistants for these actions and storing them in the
source tree of the project, where they'll be automatically picked up by
DevAssistant. It is my plan to push for custom (mostly prep and tweak)
assistants stored in source trees rather than DAPs only, because it
improves the acceptance of assistants through reduced workload for the
developer.
If you (or anyone else) have a better idea how to do this, lay it on me,
please.
TR
5:14 a.m.
New subject: Counterproposal for client-server architecture
After we've discussed this with Tomáš, I'm sending my idea about how
client-server da might look.
My idea is less robust, but it requires less components, less
communication, less code...
The idea is to only have two main components: the client and the server.
Where the client is a very thin command line client, working very much
like Tomáš have described it.
The main difference I'm proposing here is that the server contains
everything else - webserver, logic, webui, polykit agent... everything.
The server would run under "joe" - i.e. under the unprivileged user
account of the developer.
There are two main scenarios:
1) Joe wants to run da from the command line
In this case, Joe will just use da as he is using it today, when he
runs a "da something" command, da will check some magic file like
~/.devassistant/server.pid to see if the server is running or not.
* If the file does not exist, it will spawn the server [1] and save
the PID, port, etc. to that file, and client connects to it
* If the file does exist and the server is not actually running,
the file is deleted and the previous thing happens
* If the file exists and the server is running, client connects to it
(In case the server is too much resources consuming, it can
terminate itself after X minutes of inactivity, or something like that.)
2) Joe wants to use the WebUI
In this case Joe will run something like:
da server run # the exact command is not important right now
And da will do the same thing as in 1, but this time there will be
no connecting, instead a link for Joe will be printed.
It will also be possible to stop the server by something like:
da server stop
I think this proposal will work when installing form PyPI, when
installing from RPM and even when installing on a different platform
(the spawning might start the designated WM, etc.).
I see the following advantages:
* we will not need to run everything trough (client <-HTTP-> webserver
<-dbus-> bridge [<-dbus-> agent]) pipeline
* the code will be easier to implement using as much as possible from
the current code base
* the entire thing would be easier to run for the user
* no need for extra user accounts
I see the following disadvantages:
* there will be no permanent <1000 da server port, but instead new
1000 ports will be assigned as available
* each user on a
machine will get their own webserver running (but
frankly, who uses multiple user logins at the same time for developing?)
* we still need to somehow check if the user who is accessing the WebUI
is the one who launched it (but this is also the case of the previous
proposal)
[1]
http://stackoverflow.com/questions/6011235/run-a-program-from-python-and-...
1:53 a.m.
New subject: Counterproposal for client-server architecture
Hi Miro,
sorry for the late response.
Generally, I'm not against simplifying the architecture that Tomas proposed, but I see
a problem with your idea.
In the proposed architecture, it should be possible to deploy DA to a company server,
where all developers of the given company would use it. In your proposal, this would mean
running a DA server instance for every developer - this could be very costly (in terms of
HW resources). In Tomas's proposal, this only means running one DA server instance and
spinning bridge instances on demand.
I talked to Tomas and he said that he'd incorporate some of your thoughts into his
proposal. So let's wait for Tomas's update and then have one more round of
discussion about it.
Thanks a lot!
Slavek
----- Original Message -----
After we've discussed this with Tomáš, I'm sending my idea
about how
client-server da might look.
My idea is less robust, but it requires less components, less
communication, less code...
The idea is to only have two main components: the client and the server.
Where the client is a very thin command line client, working very much
like Tomáš have described it.
The main difference I'm proposing here is that the server contains
everything else - webserver, logic, webui, polykit agent... everything.
The server would run under "joe" - i.e. under the unprivileged user
account of the developer.
There are two main scenarios:
1) Joe wants to run da from the command line
In this case, Joe will just use da as he is using it today, when he
runs a "da something" command, da will check some magic file like
~/.devassistant/server.pid to see if the server is running or not.
* If the file does not exist, it will spawn the server [1] and save
the PID, port, etc. to that file, and client connects to it
* If the file does exist and the server is not actually running,
the file is deleted and the previous thing happens
* If the file exists and the server is running, client connects to it
(In case the server is too much resources consuming, it can
terminate itself after X minutes of inactivity, or something like that.)
2) Joe wants to use the WebUI
In this case Joe will run something like:
da server run # the exact command is not important right now
And da will do the same thing as in 1, but this time there will be
no connecting, instead a link for Joe will be printed.
It will also be possible to stop the server by something like:
da server stop
I think this proposal will work when installing form PyPI, when
installing from RPM and even when installing on a different platform
(the spawning might start the designated WM, etc.).
I see the following advantages:
* we will not need to run everything trough (client <-HTTP-> webserver
<-dbus-> bridge [<-dbus-> agent]) pipeline
* the code will be easier to implement using as much as possible from
the current code base
* the entire thing would be easier to run for the user
* no need for extra user accounts
I see the following disadvantages:
* there will be no permanent <1000 da server port, but instead new
>1000 ports will be assigned as available
* each user on a machine will get their own webserver running (but
frankly, who uses multiple user logins at the same time for developing?)
* we still need to somehow check if the user who is accessing the WebUI
is the one who launched it (but this is also the case of the previous
proposal)
[1]
http://stackoverflow.com/questions/6011235/run-a-program-from-python-and-...
_______________________________________________
devassistant mailing list
devassistant(a)lists.fedoraproject.org
https://lists.fedoraproject.org/mailman/listinfo/devassistant
--
Regards,
Slavek Kabrda
3257
days inactive
3294
days old
devassistant@lists.fedoraproject.org
9 comments
3 participants
participants (3)
-
Bohuslav Kabrda -
Miro Hrončok -
Tomas Radej