Moving forward with Fedora's PDC

[Clement Verna]

Hi all,

I would like to start working on the replacement application for PDC in
Fedora. So far I have been mostly looking at the technical side of the
application and how it will be setup in our infrastructure.

Below is a list of the design decisions I am looking to take. I would
really value some feedback on this in order to see if this is realistic or
not.

# Technical stack
    * Python 3 (just making it obvious)
    * Django 1.11 LTS (supported until at least April 2020) [0]
    * DRF 3.8.2 (latest version)
Using Django and DRF (Django REST Framework) should allow us to reuse some
of PDC code.

# Application architecture
    * 1 backend service used to update the PDC database (currently
pdc-updater [1])
    * 1 frontend service serving the API HTTP endpoint and the HTML
documentation of these endpoint.
    * 1 database

# Deployment architecture
    * Backend service runs in Openshift (need access to the
fedora-messaging bus)
    * Frontend service runs in Openshift
    * Database runs in on Fedora Infrastructure database hosts.

# Development Workflow
    * I would like to have the possibility for these services to be
automatically deployed on our Openshift instances (stg and prod) using the
Github build trigger [2] available in Openshift.
An idea to achieve this is to use Openshift S2I (source-to-image) [3]
feature and use PyPi to install the application dependencies. In order to
control which dependencies we are using and pulling from PyPi we could use
devpi [4] as a caching proxy.

Thanks,
Clément

[0] - https://www.djangoproject.com/download/
[1] - https://github.com/fedora-infra/pdc-updater
[2] -
https://docs.openshift.com/online/dev_guide/builds/triggering_builds.html...
[3] - https://github.com/openshift/source-to-image
[4] - https://devpi.net/docs/devpi/devpi/latest/%2Bd/index.html

[Kevin Fenzi]

On 09/11/2018 11:48 AM, Clement Verna wrote:
>  Hi all,
>  
>  I would like to start working on the replacement application for PDC in
>  Fedora. So far I have been mostly looking at the technical side of the
>  application and how it will be setup in our infrastructure.
>  
>  Below is a list of the design decisions I am looking to take. I would
>  really value some feedback on this in order to see if this is realistic or
>  not.
>  
>  # Technical stack
>      * Python 3 (just making it obvious)
>      * Django 1.11 LTS (supported until at least April 2020) [0]
>      * DRF 3.8.2 (latest version)
>  Using Django and DRF (Django REST Framework) should allow us to reuse some
>  of PDC code.
>  
>  # Application architecture
>      * 1 backend service used to update the PDC database (currently
>  pdc-updater [1])
>      * 1 frontend service serving the API HTTP endpoint and the HTML
>  documentation of these endpoint.
>      * 1 database
>  
>  # Deployment architecture
>      * Backend service runs in Openshift (need access to the
>  fedora-messaging bus)
>      * Frontend service runs in Openshift
>      * Database runs in on Fedora Infrastructure database hosts.
>  
>  # Development Workflow
>      * I would like to have the possibility for these services to be
>  automatically deployed on our Openshift instances (stg and prod) using the
>  Github build trigger [2] available in Openshift.
>  An idea to achieve this is to use Openshift S2I (source-to-image) [3]
>  feature and use PyPi to install the application dependencies. In order to
>  control which dependencies we are using and pulling from PyPi we could use
>  devpi [4] as a caching proxy. 
This all sounds perfectly great to me... everything should be there for
this with one exception: We really want to have some check in place for
s2i so that it checks license, so we don't accidentally push out
something thats not under a open source license. This doesn't need to be
a blocker, but it would be great to get in place soon.

kevin

[Aurelien Bompard]

>  everything should be there for
>  this with one exception: We really want to have some check in place for
>  s2i so that it checks license, so we don't accidentally push out
>  something thats not under a open source license. This doesn't need to be
>  a blocker, but it would be great to get in place soon. 
I could see that as an integration test in PDC, or have a regular (or
evented) job on the devpi host that would check the licences of all
cached (and thus requested) packages.
The downside of doing it on devpi is that we won't know directly which
app has requested the nonfree package. Since all dependencies are
already available locally and the license is in the package metadata
(PKG-INFO file), a script running in the integration testsuite
wouldn't even need internet access.

I can write a POC if you want.

A.

[Clement Verna]

On Wed, 12 Sep 2018 at 09:40, Aurelien Bompard <abompard(a)fedoraproject.org&gt;
wrote:

>  > everything should be there for
>  > this with one exception: We really want to have some check in place for
>  > s2i so that it checks license, so we don't accidentally push out
>  > something thats not under a open source license. This doesn't need to be
>  > a blocker, but it would be great to get in place soon.
>
>  I could see that as an integration test in PDC, or have a regular (or
>  evented) job on the devpi host that would check the licences of all
>  cached (and thus requested) packages.
>  
I like the CI test idea, a little bit like when we tests that the code base
is pep8 compliant or the test coverage in above 90%. There are a couple of
python packages that could be useful to help with that [0] [1].

[0] https://github.com/dhatim/python-license-check
[1] https://github.com/raimon49/pip-licenses


>  The downside of doing it on devpi is that we won't know directly
> which
>  app has requested the nonfree package. Since all dependencies are
>  already available locally and the license is in the package metadata
>  (PKG-INFO file), a script running in the integration testsuite
>  wouldn't even need internet access.
>
>  I can write a POC if you want.
>
>  A.
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>

[Aurelien Bompard]

>  I like the CI test idea, a little bit like when we tests that the
> code base is pep8 compliant or the test coverage in above 90%. There are a couple of
> python packages that could be useful to help with that [0] [1].
>
>  [0] https://github.com/dhatim/python-license-check
>  [1] https://github.com/raimon49/pip-licenses 
Pretty cool, I've implemented python-license-check for
fedora-messaging, it's quite trivial:
https://github.com/fedora-infra/fedora-messaging/pull/68

A.

[Kevin Fenzi]

On 09/12/2018 03:44 AM, Aurelien Bompard wrote:
> > I like the CI test idea, a little bit like when we tests that the
> code base is pep8 compliant or the test coverage in above 90%. There are a couple of
> python packages that could be useful to help with that [0] [1].
> >
> > [0] https://github.com/dhatim/python-license-check
> > [1] https://github.com/raimon49/pip-licenses
>  
>  Pretty cool, I've implemented python-license-check for
>  fedora-messaging, it's quite trivial:
>  https://github.com/fedora-infra/fedora-messaging/pull/68 
Nice! I like this idea too... it means the issue will be found closer to
the upstream app, so it can be fixed quicker than if it got merged,
released, and pushed into openshift.

We just need to make sure all the apps we deploy that use pip/s2i use
this method.

kevin

[Randy Barlow]

On 09/12/2018 04:01 AM, Clement Verna wrote:
>  [0] https://github.com/dhatim/python-license-check 
This looks useful, thanks for sharing!

[Michal Novotny]

Hello,

could we use this?

https://github.com/PostgREST/postgrest

we could trim down PDC to just db + REST API (+ git backend syncing
through hooks?)

It's written in Haskell (!!) and it looks really like a useful piece
of technology.

We could drop Django completely and have git for any write changes, which
means we will automatically get:

- version control
- acls
- public interface

for free without any future maintenance.

Only the writeable/interesting parts of PDC would be in DistGit.

I don't know what will be stored in PDC at this point but I think
just db + REST + git would be a suitable stack for a high-level
release-engineering-like settings.

clime

P.S. I would personally be happy to help with this setup.

On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow
<bowlofeggs(a)fedoraproject.org&gt; wrote:
>
> On 09/12/2018 04:01 AM, Clement Verna wrote:
> > [0] https://github.com/dhatim/python-license-check
>
> This looks useful, thanks for sharing!
>
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...

[Clement Verna]

On Mon, 24 Sep 2018 at 11:30, Michal Novotny <clime(a)redhat.com&gt; wrote:

>  Hello,
>
>  could we use this?
>
>  https://github.com/PostgREST/postgrest
>  
While I don't have a strong preference for one or the other, the main
reason we decided to use DRF is to be able to reuse some of code from PDC (
https://github.com/product-definition-center/product-definition-center). We
are currently working on the releases endpoint (
https://github.com/fedora-infra/fpdc) and to be fair we did not reuse to
much code so far, but I expect that it will be the case when working on
components and composes endpoint.



>  we could trim down PDC to just db + REST API (+ git backend syncing
>  through hooks?)
>
>  It's written in Haskell (!!) and it looks really like a useful piece
>  of technology.
>
>  We could drop Django completely and have git for any write changes, which
>  means we will automatically get:
>
>  - version control
>  - acls
>  - public interface
>
>  for free without any future maintenance.
>
>  Only the writeable/interesting parts of PDC would be in DistGit.
>
>  I don't know what will be stored in PDC at this point but I think
>  just db + REST + git would be a suitable stack for a high-level
>  release-engineering-like settings.
>
>  clime
>
>  P.S. I would personally be happy to help with this setup.
>
>  On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow
>  <bowlofeggs(a)fedoraproject.org&gt; wrote:
>  >
>  > On 09/12/2018 04:01 AM, Clement Verna wrote:
>  > > [0] https://github.com/dhatim/python-license-check
>  >
>  > This looks useful, thanks for sharing!
>  >
>  > _______________________________________________
>  > infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  > To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to
>  infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>

[Michal Novotny]

On Mon, Sep 24, 2018 at 2:43 PM Clement Verna <cverna(a)fedoraproject.org&gt; wrote:
>  On Mon, 24 Sep 2018 at 11:30, Michal Novotny <clime(a)redhat.com&gt; wrote:
> >
> > Hello,
> >
> > could we use this?
> >
> > https://github.com/PostgREST/postgrest
>
>
>  While I don't have a strong preference for one or the other, the main reason we
> decided to use DRF is to be able to reuse some of code from PDC
> (https://github.com/product-definition-center/product-definition-center). We are currently
> working on the releases endpoint (https://github.com/fedora-infra/fpdc) and to be fair we
> did not reuse to much code so far, but I expect that it will be the case when working on
> components and composes endpoint. 
I guess it mainly depends on the people doing the work (that means you
and the rest of the pack).

clime

>
>
>>
>> we could trim down PDC to just db + REST API (+ git backend syncing
>> through hooks?)
>>
>> It's written in Haskell (!!) and it looks really like a useful piece
>> of technology.
>>
>> We could drop Django completely and have git for any write changes, which
>> means we will automatically get:
>>
>> - version control
>> - acls
>> - public interface
>>
>> for free without any future maintenance.
>>
>> Only the writeable/interesting parts of PDC would be in DistGit.
>>
>> I don't know what will be stored in PDC at this point but I think
>> just db + REST + git would be a suitable stack for a high-level
>> release-engineering-like settings.
>>
>> clime
>>
>> P.S. I would personally be happy to help with this setup.
>>
>> On Wed, Sep 12, 2018 at 8:57 PM Randy Barlow
>> <bowlofeggs(a)fedoraproject.org&gt; wrote:
>> >
>> > On 09/12/2018 04:01 AM, Clement Verna wrote:
>> > > [0] https://github.com/dhatim/python-license-check
>> >
>> > This looks useful, thanks for sharing!
>> >
>> > _______________________________________________
>> > infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>> > To unsubscribe send an email to
infrastructure-leave(a)lists.fedoraproject.org
>> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
>> _______________________________________________
>> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>> To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> _______________________________________________
> infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...