📦 Advice on packaging azure-cli

[Major Hayden]

👋🏻 Hello there,

I'm eager to package Azure's CLI tools for Fedora that would allow users 
to manage their Microsoft Azure cloud infrastructure. This would help 
with CI/CD, information security, monitoring, and of course, deployments.

However, these cloud tools are a bit tricky to package. There are a few 
python components in the main repository[0] that make up the CLI itself:

   azure-cli
   azure-cli-core
   azure-cli-telemetry

Those seem fairly straightforward, but things get complicated because 
those three depend on *plenty* of little SDK components from another 
repository[1]. That repository contains over 220 SDK components that all 
release independently from the same git repository. They also have 
dependencies on some other bits of python that are already packaged for 
Fedora, such as cffi, cryptography, and PyYAML.

To make things a bit more complicated, the core CLI tools have strict 
dependencies on specific versions of the SDK components. For example:

     'azure-mgmt-datamigration~=4.1.0',
     'azure-mgmt-deploymentmanager~=0.2.0',
     'azure-mgmt-devtestlabs~=4.0',
     'azure-mgmt-dns~=8.0.0',

At first glance, that doesn't seem too bad since the versions of the 
core CLI tools and SDK versions move in tandem in the repositories. SDK 
components are constantly moving forward, but the CLI releases are tied 
to specific SDK component versions which do not change after release.

The dependency tree is fairly long[2], but it's also fairly standard 
between most of the SDKs.

I have several questions here:

   1) Should I make separate Fedora packages/specs for each CLI
      component and the SDK components? The SDK components look
      nearly identical from a packaging standpoint (no executables
      there, just libraries in each). If so, that would be about
      80-100 packages to make and maintain.

   2) Should I 'vendor' the SDK components into a single package and
      set it as a dependency of the core CLI tools?

   3) Should I dynamically generate spec files for the SDK components
      based on the requirements from the main CLI tools?

   4) Am I making this much, much more difficult than it really is? 🤦🏻‍♂️

Thanks in advance for any guidance you have. 🤗

[0] https://github.com/Azure/azure-cli
[1] https://github.com/Azure/azure-sdk-for-python
[2] https://gist.github.com/major/821e2f90c8a2b6111a42e35503caa3ad

-- 
Major Hayden

[Dusty Mabe]

On 5/25/21 8:29 AM, Major Hayden wrote:
>  👋🏻 Hello there,
>  
>  I'm eager to package Azure's CLI tools for Fedora that would allow users 
>  to manage their Microsoft Azure cloud infrastructure. This would help 
>  with CI/CD, information security, monitoring, and of course, deployments. 
I don't have much feedback for you, but just wanted to say thank you for
working on this. It's so nice when we can just pull packages from our 
repos and just start using them. A lot of cloud providers have containers
these days for their CLI tools, but I'd prefer to use the CLI tool in
an environment I've customized, rather than a container which might be
built from an environment I'm unfamiliar with.

We have the AWS and OpenStack CLIs in Fedora. Are there others? Would be
nice to have ones for gcloud, ibmcloud, digitalocean, vultr, and others
packaged.

Dusty

[Major Hayden]

On 5/25/21 7:53 AM, Dusty Mabe wrote:
>  I don't have much feedback for you, but just wanted to say thank
> you for
>  working on this. It's so nice when we can just pull packages from our
>  repos and just start using them. A lot of cloud providers have containers
>  these days for their CLI tools, but I'd prefer to use the CLI tool in
>  an environment I've customized, rather than a container which might be
>  built from an environment I'm unfamiliar with. 
You're welcome and I agree. 🤗

>  We have the AWS and OpenStack CLIs in Fedora. Are there others? Would
> be
>  nice to have ones for gcloud, ibmcloud, digitalocean, vultr, and others
>  packaged. 
We also have `golang-github-aliyun-cli` for Alibaba Cloud. It would be 
nice to get some of the other ones packaged, too. ☁

-- 
Major Hayden

[Neal Gompa]

On Tue, May 25, 2021 at 8:30 AM Major Hayden <major(a)mhtx.net&gt; wrote:
>  🏻 Hello there,
>
>  I'm eager to package Azure's CLI tools for Fedora that would allow users
>  to manage their Microsoft Azure cloud infrastructure. This would help
>  with CI/CD, information security, monitoring, and of course, deployments.
>
>  However, these cloud tools are a bit tricky to package. There are a few
>  python components in the main repository[0] that make up the CLI itself:
>
>     azure-cli
>     azure-cli-core
>     azure-cli-telemetry
>
>  Those seem fairly straightforward, but things get complicated because
>  those three depend on *plenty* of little SDK components from another
>  repository[1]. That repository contains over 220 SDK components that all
>  release independently from the same git repository. They also have
>  dependencies on some other bits of python that are already packaged for
>  Fedora, such as cffi, cryptography, and PyYAML.
>
>  To make things a bit more complicated, the core CLI tools have strict
>  dependencies on specific versions of the SDK components. For example:
>
>       'azure-mgmt-datamigration~=4.1.0',
>       'azure-mgmt-deploymentmanager~=0.2.0',
>       'azure-mgmt-devtestlabs~=4.0',
>       'azure-mgmt-dns~=8.0.0', 
That is *not* an exact version dependency. That is PEP440 equivalent
to say "that version or higher, up to the next major version":
https://www.python.org/dev/peps/pep-0440/#compatible-release

This is perfectly fine to package as separate components, and our
dependency generator handles it perfectly fine.

-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miro Hrončok]

On 25. 05. 21 14:29, Major Hayden wrote:
>  👋🏻 Hello there,
>  
>  I'm eager to package Azure's CLI tools for Fedora that would allow users to 
>  manage their Microsoft Azure cloud infrastructure. This would help with CI/CD, 
>  information security, monitoring, and of course, deployments.
>  
>  However, these cloud tools are a bit tricky to package. There are a few python 
>  components in the main repository[0] that make up the CLI itself:
>  
>     azure-cli
>     azure-cli-core
>     azure-cli-telemetry
>  
>  Those seem fairly straightforward, but things get complicated because those 
>  three depend on *plenty* of little SDK components from another repository[1]. 
>  That repository contains over 220 SDK components that all release independently 
>  from the same git repository. They also have dependencies on some other bits of 
>  python that are already packaged for Fedora, such as cffi, cryptography, and 
>  PyYAML.
>  
>  To make things a bit more complicated, the core CLI tools have strict 
>  dependencies on specific versions of the SDK components. For example:
>  
>       'azure-mgmt-datamigration~=4.1.0',
>       'azure-mgmt-deploymentmanager~=0.2.0',
>       'azure-mgmt-devtestlabs~=4.0',
>       'azure-mgmt-dns~=8.0.0',
>  
>  At first glance, that doesn't seem too bad since the versions of the core CLI 
>  tools and SDK versions move in tandem in the repositories. SDK components are 
>  constantly moving forward, but the CLI releases are tied to specific SDK 
>  component versions which do not change after release.
>  
>  The dependency tree is fairly long[2], but it's also fairly standard between 
>  most of the SDKs.
>  
>  I have several questions here:
>  
>     1) Should I make separate Fedora packages/specs for each CLI
>        component and the SDK components? The SDK components look
>        nearly identical from a packaging standpoint (no executables
>        there, just libraries in each). If so, that would be about
>        80-100 packages to make and maintain. 
The culprit of trying to maintain them in one component is that they are 
released independently. Maintaining various subpackages with different versions 
and release cycles from the same spec file is a huge PITA.

However, so is maintaining 100 packages. There is no clear win. I'd personally 
rather maintain the 100 packages with automatically generated build 
dependencies, but I would not say it is a *should*.

Have a look at https://src.fedoraproject.org/rpms/pyproject-rpm-macros

>     2) Should I 'vendor' the SDK components into a single
> package and
>        set it as a dependency of the core CLI tools? 
I'd say doing that is pretty much the same amount of work as maintaining a 
single-spec multi-package beast mentioned above, but less user friendly.

>     3) Should I dynamically generate spec files for the SDK
> components
>        based on the requirements from the main CLI tools? 
The spec files should be almost identical with just the 
name/summary/description/version changing, so I'd say this is a way to go.

>     4) Am I making this much, much more difficult than it really is?
> 🤦🏻‍♂️ 
It is already difficult.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Major Hayden]

On 5/25/21 8:01 AM, Miro Hrončok wrote:
>  The culprit of trying to maintain them in one component is that they
> are 
>  released independently. Maintaining various subpackages with different 
>  versions and release cycles from the same spec file is a huge PITA.
>  
>  However, so is maintaining 100 packages. There is no clear win. I'd 
>  personally rather maintain the 100 packages with automatically generated 
>  build dependencies, but I would not say it is a *should*. 
Thanks for helping me analyze those options.

>  Have a look at
> https://src.fedoraproject.org/rpms/pyproject-rpm-macros 
Where has this been all my life? 😱

> >    2) Should I 'vendor' the SDK components into a single
> package and
> >       set it as a dependency of the core CLI tools?
>  
>  I'd say doing that is pretty much the same amount of work as maintaining 
>  a single-spec multi-package beast mentioned above, but less user friendly. 
I was thinking the same. Thanks for confirming it.

> >    3) Should I dynamically generate spec files for the SDK
> components
> >       based on the requirements from the main CLI tools?
>  
>  The spec files should be almost identical with just the 
>  name/summary/description/version changing, so I'd say this is a way to go. 
👍🏻

> >    4) Am I making this much, much more difficult than it really
> is? 
> > 🤦🏻‍♂️
>  
>  It is already difficult. 
Challenge accepted. 😜

-- 
Major Hayden

[Miro Hrončok]

On 25. 05. 21 15:18, Major Hayden wrote:
> > Have a look at
> https://src.fedoraproject.org/rpms/pyproject-rpm-macros
>  
>  Where has this been all my life? 😱 
It is somewhat new. Let me know if you need assistance or if you find problems.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Dusty Mabe]

On 5/25/21 9:01 AM, Miro Hrončok wrote:
>  On 25. 05. 21 14:29, Major Hayden wrote: 
>  
>  The culprit of trying to maintain them in one component is that they are 
>  released independently. Maintaining various subpackages with different versions 
>  and release cycles from the same spec file is a huge PITA.
>  
>  However, so is maintaining 100 packages. There is no clear win. I'd personally 
>  rather maintain the 100 packages with automatically generated build 
>  dependencies, but I would not say it is a *should*.
>  
>  Have a look at https://src.fedoraproject.org/rpms/pyproject-rpm-macros
>   
I'm not a sophisticated packager so please take my questions here in good faith:

Are the 100 individual packages are only useful in this one case (i.e. would anyone
else use them for anything)?

If not, and the 100 individual packages versus the one large package with lots of
spec file complexity are about the same effort wise, then...

Should we consider the overhead costs (storage, compute, accounting etc) for doing
100 small versus one large package?

[Vitaly Zaitsev]

On 25.05.2021 16:27, Dusty Mabe wrote:
>  Should we consider the overhead costs (storage, compute, accounting
> etc) for doing
>  100 small versus one large package? 
Already discussed multiple times:
https://lists.fedoraproject.org/archives/list/[e-mail].o...

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Major Hayden]

On 5/25/21 7:29 AM, Major Hayden wrote:
>     1) Should I make separate Fedora packages/specs for each CLI
>        component and the SDK components? The SDK components look
>        nearly identical from a packaging standpoint (no executables
>        there, just libraries in each). If so, that would be about
>        80-100 packages to make and maintain. 
For what it's worth, I went down the road of making a big spec[0] with 
lots of sub-packages. The spec is generated by a python script and a 
jinja2 template. The entire build takes about 15-20 minutes to run since 
there are plenty of files to process.

After going this route, it's nice to bundle everything into one spec, 
but it's also difficult to maintain because I'll need to bump the entire 
package when certain sub-packages need updates. Also, testing packages 
with the "big spec" approach is extremely difficult since it involves 
changing to lots of different directories along the way. 😢

At this point, I'd love feedback.

1) Should I break these up into individual packages so they can move 
independently?

2) Should I keep going with the "big spec" approach?

If #1 seems best, then I think it makes sense to package the SDK 
components that are required by Azure's CLI first. There's an existing 
python-azure-sdk[1] package, but it uses the deprecated packaging method 
that Microsoft abandoned earlier this year.

[0] https://github.com/major/rpm-azure-cli/blob/main/python-azure.spec
[1] https://src.fedoraproject.org/rpms/python-azure-sdk

-- 
Major Hayden

[Benjamin Beasley]

I personally think the individual packages approach will be easier to maintain in the long
run. (If you do go down that road, I’m planning to help review them.)

– Ben Beasley

[Major Hayden]

On 6/2/21 9:07 AM, Benjamin Beasley wrote:
>  I personally think the individual packages approach will be easier to
> maintain in the long run. (If you do go down that road, I’m planning to help review them.)
I am leaning in that direction, too. Mohamed (the current maintainer) 
and I (new co-maintainer) have been talking about it in BZ 1953789[0].

While it sure is super quick to jam everything into one spec or one 
binary RPM, I keep stumbling over tiny problems that keep popping up one 
after the other.

Thanks for offering to help review and for the continued optimism, Ben. 🤗

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1953789

-- 
Major Hayden

[Neal Gompa]

On Wed, Jun 2, 2021 at 10:30 AM Major Hayden <major(a)mhtx.net&gt; wrote:

>  On 6/2/21 9:07 AM, Benjamin Beasley wrote:
>  > I personally think the individual packages approach will be easier to
>  maintain in the long run. (If you do go down that road, I’m planning to
>  help review them.)
>
>  I am leaning in that direction, too. Mohamed (the current maintainer)
>  and I (new co-maintainer) have been talking about it in BZ 1953789[0].
>
>  While it sure is super quick to jam everything into one spec or one
>  binary RPM, I keep stumbling over tiny problems that keep popping up one
>  after the other.
>
>  Thanks for offering to help review and for the continued optimism, Ben. 🤗
>
>  
I'll help with reviews too as needed. Just ping me and I'll help!


-- 
真実はいつも一つ！/ Always, there's only one truth!