pagure migration

[Kevin Fenzi]

Greetings.

Over the last 3-4 weeks I have been trying to migrate rel-eng from the
old trac to pagure. A number of issues have complicated/thwarted this:

* The pagure side was setup with issues enabled, so we have to keep
  those while importing the old ones.

* rel-eng trac is one of the oldest and biggest trac instances we have,
  with over 6500 tickets. This meant that importing takes 4-5 hours and
  if we hit a bug it all has to be done over.

* I've hit a number of bugs in the importer that have been fixed over
  time.

The last bug I hit (last week) is a bit more difficult however:

https://pagure.io/pagure-importer/issue/84

Basically once long ago we had an email to trac gateway. This is where
someone could just email [e-mail] and it would create a
ticket. Unfortunately, this breaks something that the importer expects:
The reporter: field isn't a username, it's a email address. The only
possible way I can think of to handle this is to try and go into the
trac db and re-write these as usernames. However, there could be tons
of them, we don't know. Also mapping email to username will be very
much manual.

On top of that, after the last failed import, I forgot to revert the
tickets git repo. So, new pagure issues have been filed on top of the
old tickets in the repo, meaning their numbers jumped to after the trac
ones, etc.

So, we have several ways forward here:

1. Spam all the still open tickets at the trac instance and tell them
if the issue still exists to please re-file it at pagure. Then after
some time just close the trac instance and move on.

2. Revert the tickets repo again, manually try and fix the trac db to
not have email addresses in reporter (or I guess make pagure-importer
just replace all those with 'nobody' or something). Then re-import.
This means we loose all the tickets filed since (3-4 currently).

3. Some brilliant plan from someone else....

Thoughts?

kevin

[Pierre-Yves Chibon]

On Mon, Nov 28, 2016 at 12:22:14PM -0700, Kevin Fenzi wrote:
> Greetings.
>
> Over the last 3-4 weeks I have been trying to migrate rel-eng from the
> old trac to pagure. A number of issues have complicated/thwarted this:
>
> The pagure side was setup with issues enabled, so we have to keep
> those while importing the old ones.
>
> rel-eng trac is one of the oldest and biggest trac instances we have,
> with over 6500 tickets. This meant that importing takes 4-5 hours and
> if we hit a bug it all has to be done over.
>
> I've hit a number of bugs in the importer that have been fixed over
> time.
>
> The last bug I hit (last week) is a bit more difficult however:
>
> https://pagure.io/pagure-importer/issue/84
>
> Basically once long ago we had an email to trac gateway. This is where
> someone could just email [e-mail] and it would create a
> ticket. Unfortunately, this breaks something that the importer expects:
> The reporter: field isn't a username, it's a email address. The only
> possible way I can think of to handle this is to try and go into the
> trac db and re-write these as usernames. However, there could be tons
> of them, we don't know. Also mapping email to username will be very
> much manual.

Where these emails ever linked to FAS accounts or do we not know?
> On top of that, after the last failed import, I forgot to revert the
> tickets git repo. So, new pagure issues have been filed on top of the
> old tickets in the repo, meaning their numbers jumped to after the trac
> ones, etc.
>
> So, we have several ways forward here:
> Spam all the still open tickets at the trac instance and tell them
>
> if the issue still exists to please re-file it at pagure. Then after
> some time just close the trac instance and move on.
> Revert the tickets repo again, manually try and fix the trac db to
>
> not have email addresses in reporter (or I guess make pagure-importer
> just replace all those with 'nobody' or something). Then re-import.
> This means we loose all the tickets filed since (3-4 currently).
> Some brilliant plan from someone else....

We could keep the json of the new issues added and re-add them after doing 2,
eventually correcting them for their id number.

Pierre

[Kevin Fenzi]

On Tue, 29 Nov 2016 22:25:36 +0100
Pierre-Yves Chibon  wrote:
> On Mon, Nov 28, 2016 at 12:22:14PM -0700, Kevin Fenzi wrote:
> > Greetings.
> >
> > Over the last 3-4 weeks I have been trying to migrate rel-eng from
> > the old trac to pagure. A number of issues have
> > complicated/thwarted this:
> >
> > The pagure side was setup with issues enabled, so we have to keep
> > those while importing the old ones.
> >
> > rel-eng trac is one of the oldest and biggest trac instances we
> >
> > have, with over 6500 tickets. This meant that importing takes 4-5
> > hours and if we hit a bug it all has to be done over.
> > I've hit a number of bugs in the importer that have been fixed
> >
> > over time.
> >
> > The last bug I hit (last week) is a bit more difficult however:
> >
> > https://pagure.io/pagure-importer/issue/84
> >
> > Basically once long ago we had an email to trac gateway. This is
> > where someone could just email [e-mail] and it
> > would create a ticket. Unfortunately, this breaks something that
> > the importer expects: The reporter: field isn't a username, it's a
> > email address. The only possible way I can think of to handle this
> > is to try and go into the trac db and re-write these as usernames.
> > However, there could be tons of them, we don't know. Also mapping
> > email to username will be very much manual.
>
> Where these emails ever linked to FAS accounts or do we not know?

We don't know really. Some of them likely were, some of them are
probibly spam.

Look at this trac query (for < in reporter name):

https://fedorahosted.org/rel-eng/query?reporter=~%3C&col=id&col=summ...

So we have some where reporter is:

Jesse Keating  which is jkeating in fas.

but we also have some like: 
Eu VIAGRA ® 
or
"Amazon.com Email Subscriptions"

but it looks like there's only a few spams.

So perhaps we could strip out the < > part and try and look that up as
a email in fas and if that fails just default it to 'nobody' or
something?

There's 109 of these, perhaps we could fix them on the trac side?
> > On top of that, after the last failed import, I forgot to revert the
> > tickets git repo. So, new pagure issues have been filed on top of
> > the old tickets in the repo, meaning their numbers jumped to after
> > the trac ones, etc.
> >
> > So, we have several ways forward here:
> > Spam all the still open tickets at the trac instance and tell
> >
> > them if the issue still exists to please re-file it at pagure. Then
> > after some time just close the trac instance and move on.
> > Revert the tickets repo again, manually try and fix the trac db
> >
> > to not have email addresses in reporter (or I guess make
> > pagure-importer just replace all those with 'nobody' or something).
> > Then re-import. This means we loose all the tickets filed since
> > (3-4 currently).
> > Some brilliant plan from someone else....
>
> We could keep the json of the new issues added and re-add them after
> doing 2, eventually correcting them for their id number.

Sure, if thats possible that would be great.

kevin