3:58 p.m.
Hello Pagure devel list,
Sorry if this isn't quite the right place for this question.
The FSF is looking into hosting our own Web based SCM service, possibly
using Pagure. We would like to how the pagure.io team handles spam,
abuse, and overly large repos on your site. We expect that this could be
an issue, and that it would be critical for keeping our new site alive
and well. Any input on this topic would be very helpful.
Thanks, : )
Andrew
2:40 a.m.
On Wed, Dec 04, 2019 at 04:58:20PM -0500, Andrew Engelbrecht wrote:
Hello Pagure devel list,
Sorry if this isn't quite the right place for this question.
The FSF is looking into hosting our own Web based SCM service, possibly
using Pagure. We would like to how the pagure.io team handles spam,
abuse, and overly large repos on your site. We expect that this could be
an issue, and that it would be critical for keeping our new site alive
and well. Any input on this topic would be very helpful.
Hi Andrew,
Thank you for your interest in pagure.
We have not had to much spam on pagure.io yet (or it hasn't been reported), but
pagure has a mechanism for admin to block an user for a definite amount of time
(from a day to a century or more) so we've used that mechanism in the few cases
of spam we dealt with.
For overly large repo, we've mostly tested with the kernel. Pagure is kinda slow
then but it works. Issue tracking should not be an issue, PRs may be fairly
slow.
There is a staging instance at https://stg.pagure.io if you want to play with
pagure without breaking pagure.io (but please let us know if you break that
instance :)).
Hoping this helps,
Pierre
10:18 a.m.
On Wed, Dec 4, 2019 at 4:58 PM Andrew Engelbrecht <andrew(a)fsf.org> wrote:
Hello Pagure devel list,
Sorry if this isn't quite the right place for this question.
The FSF is looking into hosting our own Web based SCM service, possibly
using Pagure. We would like to how the pagure.io team handles spam,
abuse, and overly large repos on your site. We expect that this could be
an issue, and that it would be critical for keeping our new site alive
and well. Any input on this topic would be very helpful.
There are two issues with "large" Pagure systems that you might be
concerned with:
* Gitolite configuration regeneration is slow with lots of
repositories. In the default configuration, Gitolite is the backend
used for managing Git ACLs. Pagure has an alternative integrated
backend that is much more performant. This will likely be better for
heavier instances. We are exploring changing Pagure's default from
Gitolite to the integrated backend to simplify
installation/configuration and improve performance. The Pagure servers
run by Fedora (src.fedoraproject.org, git.centos.org, pagure.io) don't
use Gitolite anymore for this reason. This is configurable in
/etc/pagure/pagure.cfg (as installed by the Fedora, Mageia, or
openSUSE packages) as documented[1].
* Very large repos tend to be mostly fine in Pagure. The slowest
actions are currently listing commits and computing the various stats.
This is because that data is more-or-less processed on demand, and
really large repos like the Linux kernel tend to be a struggle to
process on-demand. Even GitHub and GitLab struggle with some of this,
the difference is that we don't have any background tasks that do this
and cache the results regularly, so it's more obvious. It does work
and doesn't cause the backend to die, it just takes longer than your
average repository to compute them.
I also found this page about the forge evaluations on LibrePlanet[2],
and I think I can provide some answers to your questions there (as I
run a few Pagure instances myself and I'm the maintainer of the pagure
package in Fedora, Mageia, and openSUSE).
There is only a top level namespace, so there might be some conflicts
in terms of who reserves a repo name first. With the import/export functionality, at least
a fork does not need to be stuck under /fork.
This is actually configurable. You can set this in
/etc/pagure/pagure.cfg with the USER_NAMESPACE option as
documented[3].
Very bad usability with no js from site (gnu criteria A): login,
logout, create issue etc. Would need to investigate adding functionality without js. Could
package js into an extension, but this is not ideal for users.
At least for login and logout, this depends on the auth backend used.
Fedora's instances all leverage OpenID Connect through Ipsilon[4], and
our theme does include JavaScript. You could just as easily support a
JS-free login with Ipsilon. Or, if you want to use Pagure's local auth
backend, there is no JavaScript in the register/login/logout flow that
I recall. As for issues/PRs/etc. it would be possible to restore
JS-free usability. This is an RFE for this[5], but no work has been
done. Pull requests to improve this are welcome!
Ease of deployment, upgrading, debugging
The basic setup of Pagure is pretty easy. I've documented quick-start
processes in the Fedora, Mageia, and openSUSE packages[6][7][8] that
do reliably work. These are just simplified versions of pingou's blog
post about setting up Pagure on a Banana Pi[9], adapted for the distro
and the newest versions of Pagure. The pagure documentation does a
good job of explaining deployment strategies[10] in better detail.
Upgrade procedures are mostly just update the files and restart the
services. In cases where database migrations or other things are
required, they are documented[11].
Resource requirements / performance under load
A heavily loaded system will want to probably configure celery to have
separate queues for processing various types of tasks[12]. This will
help mitigate issues with the user experience. In addition, you will
want to have the Redis server on a separate server[13] and a remote
SQL server[14] to eliminate the contention between Pagure and the
database servers. Fedora uses a separate PostgreSQL for its instances,
while CentOS uses a separate MariaDB, so either works. I personally
use PostgreSQL. You may also want to consider using
repoSpanner[15][16] to have distributed storage for Git. This is a
somewhat extreme option, but it is an option for scaling out Pagure.
How featureful are is are administration interface/tools (GUI or CLI)
The pagure-admin CLI tool is the primary administrative interface, and
provides a relatively intuitive way to do administrative actions for a
Pagure server. I honestly don't have to use it much, but the
admin-centric functionality is present there.
Hopefully this additional information helps!
[1]: https://docs.pagure.org/pagure/configuration.html#git-auth-backend
[2]: https://libreplanet.org/wiki/Fsf_2019_forge_evaluation
[3]: https://docs.pagure.org/pagure/configuration.html#user-namespace
[4]: https://ipsilon-project.org/
[5]: https://pagure.io/pagure/issue/3507
[6]: https://src.fedoraproject.org/rpms/pagure/blob/master/f/pagure-README.Fedora
[7]:
http://svnweb.mageia.org/packages/cauldron/pagure/current/SOURCES/pagure-...
[8]:
https://build.opensuse.org/package/view_file/openSUSE:Factory/pagure/pagu...
[9]: http://blog.pingoured.fr/index.php?post/2016/01/05/Setting-up-pagure-on-a...
[10]: https://docs.pagure.org/pagure/install.html
[11]: https://pagure.io/pagure/blob/master/f/UPGRADING.rst
[12]: https://docs.pagure.org/pagure/configuration.html#celery-queue-options
[13]: https://docs.pagure.org/pagure/configuration.html#redis-options
[14]: https://docs.pagure.org/pagure/configuration.html#db-url
[15]: https://github.com/repoSpanner/repoSpanner
[16]: https://docs.pagure.org/pagure/configuration.html#repospanner-options
--
真実はいつも一つ！/ Always, there's only one truth!
11:20 a.m.
On 12/28/19 11:18 AM, Neal Gompa wrote:
There are two issues with "large" Pagure systems that you
might be
concerned with:
I also found this page about the forge evaluations on
LibrePlanet[2],
and I think I can provide some answers to your questions there (as I
run a few Pagure instances myself and I'm the maintainer of the pagure
package in Fedora, Mageia, and openSUSE).
Hello Neal and Pierre-Yves,
Thanks very much for your help. Pagure is looking like a better option
now. : )
Thanks,
Andrew
6:28 p.m.
On Mon, Dec 30, 2019 at 12:20 PM Andrew Engelbrecht <andrew(a)fsf.org> wrote:
On 12/28/19 11:18 AM, Neal Gompa wrote:
> There are two issues with "large" Pagure systems that you might be
> concerned with:
> I also found this page about the forge evaluations on LibrePlanet[2],
> and I think I can provide some answers to your questions there (as I
> run a few Pagure instances myself and I'm the maintainer of the pagure
> package in Fedora, Mageia, and openSUSE).
Hello Neal and Pierre-Yves,
Thanks very much for your help. Pagure is looking like a better option
now. : )
Awesome. I'm glad to hear that the information helped!
I don't know if you've seen it, but Pierre-Yves and I gave a talk at
the openSUSE Conference back in May introducing Pagure:
https://media.ccc.de/v/2432-introducing-pagure
It might also help for you to know because of how Pagure "projects"
are stored (they are a collection of Git repositories), it's also
possible to support fully local management of a project using tools
like pag-off[1]. This would probably be useful for folks who don't
like primarily using the web for project management.
The nature of Pagure projects is leveraged for handling importing from
other systems into Pagure. Tools like pagure-importer[2] take
advantage of the defined data handling to craft it from other hosting
systems (like GitHub or Trac) for importing into Pagure. It could also
be theoretically used to pull in project data from various services
(Savane, Bugzilla, etc.) to unify them under a Pagure project. Now, we
don't have code for that, but the possibility is there.
There's also a simple Python library for interfacing with Pagure
instances called libpagure[3], which people can use to build scripts
and tools for supporting their own workflows.
Finally, every Pagure instance has an /api endpoint which renders a
web page documenting the API it provides for people to use with their
own code. You can see this for pagure.io on https://pagure.io/api.
There are also integrations for FOSS software solutions for Pagure.
Pagure itself includes support for Jenkins[4]. Zuul CI has a driver
for Pagure as well[5]. The Weblate FOSS translation platform also
integrates with Pagure[6]. The pagure.io system has Jenkins connected,
and src.fedoraproject.org is connected to a Zuul CI instance. Fedora
is in the process of getting a Weblate instance set up and connecting
it to our Pagure instances. Pagure also has a well-defined webhook
system to connect it with other services[7].
So insofar as supporting a Free Software forge with a wide array of
functionality, there's a growing ecosystem around Pagure that can help
make your forge a very appealing option for projects.
[1]: https://pagure.io/pag-off
[2]: https://pagure.io/pagure-importer
[3]: https://pagure.io/libpagure
[4]: https://docs.pagure.org/pagure/usage/pagure_ci.html
[5]: https://zuul-ci.org/docs/zuul/admin/drivers/pagure.html
[6]:
https://docs.weblate.org/en/latest/admin/continuous.html#automatically-re...
[7]: https://docs.pagure.org/pagure/usage/using_webhooks.html
--
真実はいつも一つ！/ Always, there's only one truth!
3:27 a.m.
19/12/28 17:18(e)an, Neal Gompa igorleak idatzi zuen:
On Wed, Dec 4, 2019 at 4:58 PM Andrew Engelbrecht
<andrew(a)fsf.org> wrote:
>
> Hello Pagure devel list,
>
> Sorry if this isn't quite the right place for this question.
>
> The FSF is looking into hosting our own Web based SCM service, possibly
> using Pagure. We would like to how the pagure.io team handles spam,
> abuse, and overly large repos on your site. We expect that this could be
> an issue, and that it would be critical for keeping our new site alive
> and well. Any input on this topic would be very helpful.
>
There are two issues with "large" Pagure systems that you might be
concerned with:
* Gitolite configuration regeneration is slow with lots of
repositories. In the default configuration, Gitolite is the backend
used for managing Git ACLs. Pagure has an alternative integrated
backend that is much more performant. This will likely be better for
heavier instances. We are exploring changing Pagure's default from
Gitolite to the integrated backend to simplify
installation/configuration and improve performance. The Pagure servers
run by Fedora (src.fedoraproject.org, git.centos.org, pagure.io) don't
use Gitolite anymore for this reason. This is configurable in
/etc/pagure/pagure.cfg (as installed by the Fedora, Mageia, or
openSUSE packages) as documented[1].
* Very large repos tend to be mostly fine in Pagure. The slowest
actions are currently listing commits and computing the various stats.
This is because that data is more-or-less processed on demand, and
really large repos like the Linux kernel tend to be a struggle to
process on-demand. Even GitHub and GitLab struggle with some of this,
the difference is that we don't have any background tasks that do this
and cache the results regularly, so it's more obvious. It does work
and doesn't cause the backend to die, it just takes longer than your
average repository to compute them.
I also found this page about the forge evaluations on LibrePlanet[2],
and I think I can provide some answers to your questions there (as I
run a few Pagure instances myself and I'm the maintainer of the pagure
package in Fedora, Mageia, and openSUSE).
> There is only a top level namespace, so there might be some conflicts in terms of who
reserves a repo name first. With the import/export functionality, at least a fork does not
need to be stuck under /fork.
This is actually configurable. You can set this in
/etc/pagure/pagure.cfg with the USER_NAMESPACE option as
documented[3].
> Very bad usability with no js from site (gnu criteria A): login, logout, create issue
etc. Would need to investigate adding functionality without js. Could package js into an
extension, but this is not ideal for users.
At least for login and logout, this depends on the auth backend used.
Fedora's instances all leverage OpenID Connect through Ipsilon[4], and
our theme does include JavaScript. You could just as easily support a
JS-free login with Ipsilon. Or, if you want to use Pagure's local auth
backend, there is no JavaScript in the register/login/logout flow that
I recall. As for issues/PRs/etc. it would be possible to restore
JS-free usability. This is an RFE for this[5], but no work has been
done. Pull requests to improve this are welcome!
I've tested /login /logout on all auth backend during our CSP hardening
process during last july and iirc , openid|fas is the unique auth
backend that uses js during the login process to hide a redirection
form, but is not actually necessary: it's just a visual thing
>
>> Ease of deployment, upgrading, debugging
>
> The basic setup of Pagure is pretty easy. I've documented quick-start
> processes in the Fedora, Mageia, and openSUSE packages[6][7][8] that
> do reliably work. These are just simplified versions of pingou's blog
> post about setting up Pagure on a Banana Pi[9], adapted for the distro
> and the newest versions of Pagure. The pagure documentation does a
> good job of explaining deployment strategies[10] in better detail.
> Upgrade procedures are mostly just update the files and restart the
> services. In cases where database migrations or other things are
> required, they are documented[11].
>
>> Resource requirements / performance under load
>
> A heavily loaded system will want to probably configure celery to have
> separate queues for processing various types of tasks[12]. This will
> help mitigate issues with the user experience. In addition, you will
> want to have the Redis server on a separate server[13] and a remote
> SQL server[14] to eliminate the contention between Pagure and the
> database servers. Fedora uses a separate PostgreSQL for its instances,
> while CentOS uses a separate MariaDB, so either works. I personally
> use PostgreSQL. You may also want to consider using
> repoSpanner[15][16] to have distributed storage for Git. This is a
> somewhat extreme option, but it is an option for scaling out Pagure.
>
>> How featureful are is are administration interface/tools (GUI or CLI)
>
> The pagure-admin CLI tool is the primary administrative interface, and
> provides a relatively intuitive way to do administrative actions for a
> Pagure server. I honestly don't have to use it much, but the
> admin-centric functionality is present there.
>
> Hopefully this additional information helps!
>
>
> [1]: https://docs.pagure.org/pagure/configuration.html#git-auth-backend
> [2]: https://libreplanet.org/wiki/Fsf_2019_forge_evaluation
> [3]: https://docs.pagure.org/pagure/configuration.html#user-namespace
> [4]: https://ipsilon-project.org/
> [5]: https://pagure.io/pagure/issue/3507
> [6]: https://src.fedoraproject.org/rpms/pagure/blob/master/f/pagure-README.Fedora
> [7]:
http://svnweb.mageia.org/packages/cauldron/pagure/current/SOURCES/pagure-...
> [8]:
https://build.opensuse.org/package/view_file/openSUSE:Factory/pagure/pagu...
> [9]:
http://blog.pingoured.fr/index.php?post/2016/01/05/Setting-up-pagure-on-a...
> [10]: https://docs.pagure.org/pagure/install.html
> [11]: https://pagure.io/pagure/blob/master/f/UPGRADING.rst
> [12]: https://docs.pagure.org/pagure/configuration.html#celery-queue-options
> [13]: https://docs.pagure.org/pagure/configuration.html#redis-options
> [14]: https://docs.pagure.org/pagure/configuration.html#db-url
> [15]: https://github.com/repoSpanner/repoSpanner
> [16]: https://docs.pagure.org/pagure/configuration.html#repospanner-options
>
>
1568
days inactive
1602
days old
5 comments
4 participants
participants (4)
-
Andrew Engelbrecht -
Julen Landa Alustiza -
Neal Gompa -
Pierre-Yves Chibon