RamaLama is an open source competitor to Ollama. The goal is to make the use of AI Models as simple as Podman or Docker. But able to support any AI Model registry. HuggingFace, Ollama as well as OCI Registries (quay.io, docker hug, artifactory ...)
It uses either Podman or Docker under the hood to run your AI Models in containers, but can also run containers natively on the host.
We are looking for contributors in any form, but really could use some help getting it packaged for Fedora, PyPy and Brew for Macs.
We have setup a discord room for discussions on RamaLama. https://t.co/wdJ2KWJ9de https://t.co/wdJ2KWJ9de
The code is all written in Python.
Join the initiative to make running Open Source AI Models simple and boring.
On Mon, 2024-09-30 at 11:18 -0400, Daniel Walsh wrote:
RamaLama is an open source competitor to Ollama. The goal is to make the use of AI Models as simple as Podman or Docker. But able to support any AI Model registry. HuggingFace, Ollama as well as OCI Registries (quay.io, docker hug, artifactory ...)
It uses either Podman or Docker under the hood to run your AI Models in containers, but can also run containers natively on the host.
We are looking for contributors in any form, but really could use some help getting it packaged for Fedora, PyPy and Brew for Macs.
We have setup a discord room for discussions on RamaLama. https://t.co/wdJ2KWJ9de https://t.co/wdJ2KWJ9de
The code is all written in Python.
Join the initiative to make running Open Source AI Models simple and boring.
Having a quick look at it...I assume for packaging purposes we should avoid that yoiks-inducing `install.py` like the plague? Is the setup.py file sufficient to install it properly in a normal way? On the face of it it doesn't look like it would be, but maybe I'm missing something. Given that we're in the 2020s, why doesn't it have a pyproject.toml ?
Thanks!
On Mon, 2024-09-30 at 09:24 -0700, Adam Williamson wrote:
On Mon, 2024-09-30 at 11:18 -0400, Daniel Walsh wrote:
RamaLama is an open source competitor to Ollama. The goal is to make the use of AI Models as simple as Podman or Docker. But able to support any AI Model registry. HuggingFace, Ollama as well as OCI Registries (quay.io, docker hug, artifactory ...)
It uses either Podman or Docker under the hood to run your AI Models in containers, but can also run containers natively on the host.
We are looking for contributors in any form, but really could use some help getting it packaged for Fedora, PyPy and Brew for Macs.
We have setup a discord room for discussions on RamaLama. https://t.co/wdJ2KWJ9de https://t.co/wdJ2KWJ9de
The code is all written in Python.
Join the initiative to make running Open Source AI Models simple and boring.
Having a quick look at it...I assume for packaging purposes we should avoid that yoiks-inducing `install.py` like the plague? Is the setup.py file sufficient to install it properly in a normal way? On the face of it it doesn't look like it would be, but maybe I'm missing something. Given that we're in the 2020s, why doesn't it have a pyproject.toml ?
Thanks!
Erf...and then ramalama.py goes to the trouble of adding the non- standard directory the Python lib was installed to the path before importing it:
https://github.com/containers/ramalama/blob/main/ramalama.py#L10-L15
why all this? Why not just have it set up as a perfectly normal Python lib+script project and let all the infrastructure Python-world has been building for decades handle installing it on various platforms? Is there something I'm missing here, or should I send a PR?
Is it because this was written for macOS originally? But surely there's a standard way to install a normal Python project on macOS that doesn't require a custom install script?!
On 9/30/24 12:24, Adam Williamson wrote:
On Mon, 2024-09-30 at 11:18 -0400, Daniel Walsh wrote:
RamaLama is an open source competitor to Ollama. The goal is to make the use of AI Models as simple as Podman or Docker. But able to support any AI Model registry. HuggingFace, Ollama as well as OCI Registries (quay.io, docker hug, artifactory ...)
It uses either Podman or Docker under the hood to run your AI Models in containers, but can also run containers natively on the host.
We are looking for contributors in any form, but really could use some help getting it packaged for Fedora, PyPy and Brew for Macs.
We have setup a discord room for discussions on RamaLama. https://t.co/wdJ2KWJ9de https://t.co/wdJ2KWJ9de
The code is all written in Python.
Join the initiative to make running Open Source AI Models simple and boring.
Having a quick look at it...I assume for packaging purposes we should avoid that yoiks-inducing `install.py` like the plague? Is the setup.py file sufficient to install it properly in a normal way? On the face of it it doesn't look like it would be, but maybe I'm missing something. Given that we're in the 2020s, why doesn't it have a pyproject.toml ?
Thanks!
install.py is just for installing from a web site.
make install
Should do everything required for setting up the install. The current rpm/python-ramalama.spec
uses a combination of pyproject_install and make install to put files in place.
%install %pyproject_install %pyproject_save_files %{pypi_name} %{__make} DESTDIR=%{buildroot} PREFIX=%{_prefix} install-shortnames %{__make} DESTDIR=%{buildroot} PREFIX=%{_prefix} install-docs %{__make} DESTDIR=%{buildroot} PREFIX=%{_prefix} install-completions
Not sure if you can do everything in python setup.py files. Shortnames, man pages (install-docs) and command completion files are handled separately.
pyproject.toml is new to me.
Lokesh worked on some of this with limited time.
On 9/30/24 12:33, Adam Williamson wrote:
On Mon, 2024-09-30 at 09:24 -0700, Adam Williamson wrote:
On Mon, 2024-09-30 at 11:18 -0400, Daniel Walsh wrote:
RamaLama is an open source competitor to Ollama. The goal is to make the use of AI Models as simple as Podman or Docker. But able to support any AI Model registry. HuggingFace, Ollama as well as OCI Registries (quay.io, docker hug, artifactory ...)
It uses either Podman or Docker under the hood to run your AI Models in containers, but can also run containers natively on the host.
We are looking for contributors in any form, but really could use some help getting it packaged for Fedora, PyPy and Brew for Macs.
We have setup a discord room for discussions on RamaLama. https://t.co/wdJ2KWJ9de https://t.co/wdJ2KWJ9de
The code is all written in Python.
Join the initiative to make running Open Source AI Models simple and boring.
Having a quick look at it...I assume for packaging purposes we should avoid that yoiks-inducing `install.py` like the plague? Is the setup.py file sufficient to install it properly in a normal way? On the face of it it doesn't look like it would be, but maybe I'm missing something. Given that we're in the 2020s, why doesn't it have a pyproject.toml ?
Thanks!
Erf...and then ramalama.py goes to the trouble of adding the non- standard directory the Python lib was installed to the path before importing it:
https://github.com/containers/ramalama/blob/main/ramalama.py#L10-L15
why all this? Why not just have it set up as a perfectly normal Python lib+script project and let all the infrastructure Python-world has been building for decades handle installing it on various platforms? Is there something I'm missing here, or should I send a PR?
Is it because this was written for macOS originally? But surely there's a standard way to install a normal Python project on macOS that doesn't require a custom install script?!
We are trying to run ramalama inside of a container, this means that we are volume mounting in the directory from the host into the container along with the ramalama executable.
We want to make sure that the ramalama executable finds the python libraries ,which is why we are sticking the library into first place in the path. (It is a little hacky and we could probably work around it using environment variables rather then inserting it.
Bottom line we want the executable from the host running inside of the container, to try to avoid drift between container images and the executable.
On Mon, 2024-09-30 at 12:41 -0400, Daniel Walsh wrote:
why all this? Why not just have it set up as a perfectly normal Python lib+script project and let all the infrastructure Python-world has been building for decades handle installing it on various platforms? Is there something I'm missing here, or should I send a PR?
Is it because this was written for macOS originally? But surely there's a standard way to install a normal Python project on macOS that doesn't require a custom install script?!
We are trying to run ramalama inside of a container, this means that we are volume mounting in the directory from the host into the container along with the ramalama executable.
We want to make sure that the ramalama executable finds the python libraries ,which is why we are sticking the library into first place in the path. (It is a little hacky and we could probably work around it using environment variables rather then inserting it.
Bottom line we want the executable from the host running inside of the container, to try to avoid drift between container images and the executable.
So...the script and library are initially run from a host system, and they create a container then copy/mount themselves into it and re- execute within it? Do I have that right? ... yeah, it looks like that is what `run_container` in `ramalama/cli.py` does. ooookay.
Honestly I feel like I would've written this so the bit that runs in the container is just a different thing from the bit that runs outside of the container? But I guess that's getting rather beyond the scope of "package the thing that exists". :D
On 9/30/24 12:52, Adam Williamson wrote:
On Mon, 2024-09-30 at 12:41 -0400, Daniel Walsh wrote:
why all this? Why not just have it set up as a perfectly normal Python lib+script project and let all the infrastructure Python-world has been building for decades handle installing it on various platforms? Is there something I'm missing here, or should I send a PR?
Is it because this was written for macOS originally? But surely there's a standard way to install a normal Python project on macOS that doesn't require a custom install script?!
We are trying to run ramalama inside of a container, this means that we are volume mounting in the directory from the host into the container along with the ramalama executable.
We want to make sure that the ramalama executable finds the python libraries ,which is why we are sticking the library into first place in the path. (It is a little hacky and we could probably work around it using environment variables rather then inserting it.
Bottom line we want the executable from the host running inside of the container, to try to avoid drift between container images and the executable.
So...the script and library are initially run from a host system, and they create a container then copy/mount themselves into it and re- execute within it? Do I have that right? ... yeah, it looks like that is what `run_container` in `ramalama/cli.py` does. ooookay.
Honestly I feel like I would've written this so the bit that runs in the container is just a different thing from the bit that runs outside of the container? But I guess that's getting rather beyond the scope of "package the thing that exists". :D
Yup. Not sure it is the best and does complicate development.
BTW I opened https://github.com/containers/ramalama/pull/209 for the pyproject.toml file.
On Mon, 2024-09-30 at 12:41 -0400, Daniel Walsh wrote:
We want to make sure that the ramalama executable finds the python libraries ,which is why we are sticking the library into first place in the path. (It is a little hacky and we could probably work around it using environment variables rather then inserting it.
OK, so this is kinda fun, actually.
setuptools used to provide pkg_resources.resource_filename which you could use to get the filesystem location of files within any given module. But it got a bit weird, because these days in Python, a module doesn't have to exist as a simple file on the filesystem. It could be part of some kind of archived distribution (an egg or wheel, for instance), or imported via zipimport, or something. pkg_resources did some fairly hairy workarounds for these cases - it would basically dump the module out to a temporary cache on disk and give you *that* location, see https://setuptools.pypa.io/en/latest/pkg_resources.html#resource-extraction .
That mess is now deprecated and we have `importlib.resources` instead, which doesn't pretend it can always give you a filesystem location for stuff, but just gives you a file-like interface. pkg_resources is entirely removed in recent Python (3.12 or 3.13, I forget which).
So...looked at a certain way, yeah, I suppose you *can* say it makes sense for ramalama to have a custom install process simply to ensure the library *is* actually on disk to be mounted into the container. If we did things the 'proper' way and used importlib.resources , we'd have to sort of do what pkg_resources used to do: use the importlib interface to dump the contents of all the files in the library to a known temporary filesystem location and mount *that* into the container. (Arguably, that *still* might be better, but eh.)
The overall design still feels a bit weird to me, though. It feels like there should just be overall a better way to implement "have a Python project create a container then drive stuff using a Python project inside the container" than having them both be the *same thing*?
On Mon, 2024-09-30 at 12:56 -0400, Daniel Walsh wrote:
Honestly I feel like I would've written this so the bit that runs in the container is just a different thing from the bit that runs outside of the container? But I guess that's getting rather beyond the scope of "package the thing that exists". :D
Yup. Not sure it is the best and does complicate development.
BTW I opened https://github.com/containers/ramalama/pull/209 for the pyproject.toml file.
So I thought about this a bit and had two ideas:
1) Have ramalama always do most of its work outside of any container, and when used in container-y mode, only do 'real stuff' in the container. As what ramalama ultimately *does* in the end is dispatch shell commands (unless I missed anything else it does?), this would mean having all the business logic happen on the host, but once it's actually constructed a shell command to run that will "do real stuff", it would run it inside the container. There would not need to be a copy of *ramalama per se* inside the container. In non-container mode of course it would just run the command directly.
2) Make the main meat of ramalama a script called rl-direct, or whatever, which does all the actual work, and always does it "directly" with no container involved. Then have an optional wrap-it-in-a- container script (which can still be called 'ramalama') which, when invoked, does the "create a container then run something inside the container with all the same args" thing. The thing it runs would be rl- direct inside the container. The container image would have a copy of rl-direct baked in at build time. For packaging, we would have a package with the library and the 'direct' script in it (for use directly on the host system) and a package with just the wrap-it-in-a- container script (for containerized use). The wrap-it-in-a-container script could have a --no-container arg (or whatever it is) that would cause it to just run rl-direct (assuming it was there) and pass its args through. The wrap-it-in-a-container script would not need to maintain perfect parity with the rl-direct script - all the wrapper script needs to know about is the business of setting up a container and executing a thing inside the container. All arg evaluation and so on would happen in rl-direct.
What do you think of those options? If either sounds interesting I can try and carve out some time to work on one.
On Mon, 2024-09-30 at 23:56 +0200, Miro Hrončok wrote:
On 30. 09. 24 19:22, Adam Williamson wrote:
pkg_resources is entirely removed in recent Python (3.12 or 3.13, I forget which).
pkg_resources is part of setuptools. It is deprecated, but still there.
Oh, my bad, I thought it was gone.
On 9/30/24 17:43, Adam Williamson wrote:
On Mon, 2024-09-30 at 12:56 -0400, Daniel Walsh wrote:
Honestly I feel like I would've written this so the bit that runs in the container is just a different thing from the bit that runs outside of the container? But I guess that's getting rather beyond the scope of "package the thing that exists". :D
Yup. Not sure it is the best and does complicate development.
BTW I opened https://github.com/containers/ramalama/pull/209 for the pyproject.toml file.
So I thought about this a bit and had two ideas:
- Have ramalama always do most of its work outside of any container,
and when used in container-y mode, only do 'real stuff' in the container. As what ramalama ultimately *does* in the end is dispatch shell commands (unless I missed anything else it does?), this would mean having all the business logic happen on the host, but once it's actually constructed a shell command to run that will "do real stuff", it would run it inside the container. There would not need to be a copy of *ramalama per se* inside the container. In non-container mode of course it would just run the command directly.
- Make the main meat of ramalama a script called rl-direct, or
whatever, which does all the actual work, and always does it "directly" with no container involved. Then have an optional wrap-it-in-a- container script (which can still be called 'ramalama') which, when invoked, does the "create a container then run something inside the container with all the same args" thing. The thing it runs would be rl- direct inside the container. The container image would have a copy of rl-direct baked in at build time. For packaging, we would have a package with the library and the 'direct' script in it (for use directly on the host system) and a package with just the wrap-it-in-a- container script (for containerized use). The wrap-it-in-a-container script could have a --no-container arg (or whatever it is) that would cause it to just run rl-direct (assuming it was there) and pass its args through. The wrap-it-in-a-container script would not need to maintain perfect parity with the rl-direct script - all the wrapper script needs to know about is the business of setting up a container and executing a thing inside the container. All arg evaluation and so on would happen in rl-direct.
What do you think of those options? If either sounds interesting I can try and carve out some time to work on one.
Yes we could just do the run and serve commands from within the container. Will take a look at this.
Anyways I did a lot of work and have not gotten RamaLama packaged for PyPi, so people can now do
`pip install ramalama` to try it out. Cleaned up the rpm spec file as well.
-
Adam Williamson -
Daniel Walsh -
Miro Hrončok