Hi everyone,
So it seems the last update of MM2 has fixed the issue I was seeing from time to time of the application becoming unresponsive due to DB hanging.
So here is the current status as far as I see it:
* Mirrorlist -> In prod and apparently fine * MM2 UI -> Running in stg and seems all good to me * Crawler -> Adrian Reber has put some great work on this, afaics he seems to have fixed the biggest issues but I'll let him confirm if everything works now or not (@Adrian, I know there are features you want to make to the crawler but let's get it working as it is for 1.0 and improve it afterward ;-)) * Pickle generation -> code wise I believe it works the question is: does it generate the same pickle as MM1 ! Crons: iirc there are still a couple of crons that are not running properly and returning errors
So what I would like to do at one point is: * Freeze the DB on MM1 * Generate a pickle -> save it * Sync the DB over to stg (covert it as needed) * Unfreeze the DB on MM1 * Have MM2 generate a pickle with this new data * Compare the two pickle files
This sounds to me like the easiest way of comparing if both applications behave in the same way.
Thoughts?
Pierre
On Fri, 17 Apr 2015 17:01:45 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Hi everyone,
So it seems the last update of MM2 has fixed the issue I was seeing from time to time of the application becoming unresponsive due to DB hanging.
So here is the current status as far as I see it:
- Mirrorlist -> In prod and apparently fine
- MM2 UI -> Running in stg and seems all good to me
- Crawler -> Adrian Reber has put some great work on this, afaics he
seems to have fixed the biggest issues but I'll let him confirm if everything works now or not (@Adrian, I know there are features you want to make to the crawler but let's get it working as it is for 1.0 and improve it afterward ;-))
- Pickle generation -> code wise I believe it works the question is:
does it generate the same pickle as MM1 ! Crons: iirc there are still a couple of crons that are not running properly and returning errors
So what I would like to do at one point is:
- Freeze the DB on MM1
- Generate a pickle -> save it
- Sync the DB over to stg (covert it as needed)
- Unfreeze the DB on MM1
- Have MM2 generate a pickle with this new data
- Compare the two pickle files
This sounds to me like the easiest way of comparing if both applications behave in the same way.
Thoughts?
Sounds good to me.
Do we know if the UMDL (update mirror directory list) is working as expected? If staging has the mount (and it should) can we make it scan and see if it matches up with the things production has/finds?
So, that would be another, higher level test:
Make staging scan directories for latest content. Make staging make a pkl based on that. Check it against the prod one
?
kevin
On Fri, Apr 17, 2015 at 09:05:03AM -0600, Kevin Fenzi wrote:
On Fri, 17 Apr 2015 17:01:45 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Hi everyone,
So it seems the last update of MM2 has fixed the issue I was seeing from time to time of the application becoming unresponsive due to DB hanging.
So here is the current status as far as I see it:
- Mirrorlist -> In prod and apparently fine
- MM2 UI -> Running in stg and seems all good to me
- Crawler -> Adrian Reber has put some great work on this, afaics he
seems to have fixed the biggest issues but I'll let him confirm if everything works now or not (@Adrian, I know there are features you want to make to the crawler but let's get it working as it is for 1.0 and improve it afterward ;-))
I would also say, that the crawler is ready for production. There is a one line change I would like to get into an upcoming release, but it is not really required at this point. PR will come today.
- Pickle generation -> code wise I believe it works the question is:
does it generate the same pickle as MM1 ! Crons: iirc there are still a couple of crons that are not running properly and returning errors
Which ones?
So what I would like to do at one point is:
- Freeze the DB on MM1
- Generate a pickle -> save it
- Sync the DB over to stg (covert it as needed)
- Unfreeze the DB on MM1
- Have MM2 generate a pickle with this new data
- Compare the two pickle files
This sounds to me like the easiest way of comparing if both applications behave in the same way.
Thoughts?
Sounds good to me.
Sounds also good to me.
Do we know if the UMDL (update mirror directory list) is working as expected? If staging has the mount (and it should) can we make it scan and see if it matches up with the things production has/finds?
It has the mount, but I have not verified it is working correctly. At least it does something which requires some time.
So, that would be another, higher level test:
Make staging scan directories for latest content. Make staging make a pkl based on that. Check it against the prod one
I am not sure if the resulting pickles can be that easily compared as there have been changes all around MM2 which might return slightly different results which are nevertheless both valid results.
Adrian
On Mon, Apr 20, 2015 at 12:03:20PM +0200, Adrian Reber wrote:
On Fri, Apr 17, 2015 at 09:05:03AM -0600, Kevin Fenzi wrote:
On Fri, 17 Apr 2015 17:01:45 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
So it seems the last update of MM2 has fixed the issue I was seeing from time to time of the application becoming unresponsive due to DB hanging.
So here is the current status as far as I see it:
- Mirrorlist -> In prod and apparently fine
- MM2 UI -> Running in stg and seems all good to me
- Crawler -> Adrian Reber has put some great work on this, afaics he
seems to have fixed the biggest issues but I'll let him confirm if everything works now or not (@Adrian, I know there are features you want to make to the crawler but let's get it working as it is for 1.0 and improve it afterward ;-))
I would also say, that the crawler is ready for production. There is a one line change I would like to get into an upcoming release, but it is not really required at this point. PR will come today.
- Pickle generation -> code wise I believe it works the question is:
does it generate the same pickle as MM1 ! Crons: iirc there are still a couple of crons that are not running properly and returning errors
Which ones?
From this night: ---- Subject: Cron root@mm-crawler01 /usr/bin/mm2_crawler --threads 50 > /dev/null 2>&1
/bin/sh: line 1: 18523 Killed /usr/bin/mm2_crawler --threads 50
/dev/null 2>&1
---- Subject: Cron root@mm-backend01 cd /usr/share/mirrormanager2 && /usr/bin/mm2_get_internet2_netblocks /var/lib/mirrormanager/internet2_netblocks.txt
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98. unable to retrieve netblock list. ----
These are the only two I can see right now.
So, that would be another, higher level test:
Make staging scan directories for latest content. Make staging make a pkl based on that. Check it against the prod one
I am not sure if the resulting pickles can be that easily compared as there have been changes all around MM2 which might return slightly different results which are nevertheless both valid results.
I don't think we changed the pickles themselves, for example mirrorlist barely changed and is currently working with MM1's pickle.
Pierre
On Tue, Apr 21, 2015 at 07:33:27AM +0200, Pierre-Yves Chibon wrote:
! Crons: iirc there are still a couple of crons that are not running properly and returning errors
Which ones?
From this night:
Subject: Cron root@mm-crawler01 /usr/bin/mm2_crawler --threads 50 > /dev/null 2>&1
/bin/sh: line 1: 18523 Killed /usr/bin/mm2_crawler --threads 50
/dev/null 2>&1
This is OOM. One crawler did not finish within 12 hours. The second crawler started and Linux killed one of them. The atomic tree is really killing the crawler with its over 370000 files. If a mirror has http as the only protocol and is geographically far and slow it is not possible to crawl in a reasonable time. The atomic tree needs a different setup/rule how to be crawled.
Subject: Cron root@mm-backend01 cd /usr/share/mirrormanager2 && /usr/bin/mm2_get_internet2_netblocks /var/lib/mirrormanager/internet2_netblocks.txt
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98.
[...]
gzip: /tmp/get_i2_netblocks.RS1rA07F/rib.gz: unexpected end of file short file (empty packet) at zebra-dump-parser/zebra-dump-parser.pl line 98. unable to retrieve netblock list.
I think I see these also on my MM1 installation. Probably easy to fix.
These are the only two I can see right now.
So, that would be another, higher level test:
Make staging scan directories for latest content. Make staging make a pkl based on that. Check it against the prod one
I am not sure if the resulting pickles can be that easily compared as there have been changes all around MM2 which might return slightly different results which are nevertheless both valid results.
I don't think we changed the pickles themselves, for example mirrorlist barely changed and is currently working with MM1's pickle.
I was talking about the content of the pickles. As the crawler is making slightly different decisions and umdl maybe also I do not think we can compare the content of the pickles.
Adrian
infrastructure@lists.fedoraproject.org
-
Adrian Reber -
Kevin Fenzi -
Pierre-Yves Chibon