Cleaning up tags in askfedora

[Ankur Sinha]

Hi,

I found this post on askbot that detail some management commands to
clean up tags[1].

I was thinking of running some of these as cron jobs also. What do you
folks think?

Query: does the application need to be shut down before these management
commands are run? If it does, we can maybe run it once a fortnight or
something?

I was thinking we'd first convert common tags to "standard" tags:
gnome-3 to gnome3, #fedora18 to fedora18 etc. and then get rid of unused
tags. This will reduce the clutter in the tag cloud to start with. It'll
also show correct tags in the lists and hopefully reduce further
clutter.

[1]
http://askbot.org/en/question/2673/is-there-a-way-to-purge-unused-tags-fr...
-- 
Thanks,
Warm regards,
Ankur (FranciscoD)

http://fedoraproject.org/wiki/User:Ankursinha

Join Fedora! Come talk to us!
http://fedoraproject.org/wiki/Fedora_Join_SIG

[Mathieu Bridon]

On Mon, 2013-06-10 at 22:36 +1000, Ankur Sinha wrote:
>  I was thinking we'd first convert common tags to
> "standard" tags:
>  gnome-3 to gnome3, #fedora18 to fedora18 etc. and then get rid of unused
>  tags. 
I've been doing that quite a lot for the past 2 weeks or so, and was
quite disappointed to see that tags with 0 entries still show up, and
thus encourage people to keep using them. (indeed, I've seen tags I had
cleared being used again for new questions)

Automatically deleting all tags with 0 questions associated seems like a
good thing to do, to ensure higher quality/consistency of tagging.


-- 
Mathieu

[Matthew Miller]

On Tue, Jun 11, 2013 at 10:56:13AM +0800, Mathieu Bridon wrote:
>  On Mon, 2013-06-10 at 22:36 +1000, Ankur Sinha wrote:
>  > I was thinking we'd first convert common tags to "standard" tags:
>  > gnome-3 to gnome3, #fedora18 to fedora18 etc. and then get rid of unused
>  > tags.
>  I've been doing that quite a lot for the past 2 weeks or so, and was
>  quite disappointed to see that tags with 0 entries still show up, and
>  thus encourage people to keep using them. (indeed, I've seen tags I had
>  cleared being used again for new questions)
>  Automatically deleting all tags with 0 questions associated seems like a
>  good thing to do, to ensure higher quality/consistency of tagging. 
+1. It's really a mess and this will help.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Kevin Fenzi]

On Tue, 11 Jun 2013 12:59:43 -0400
Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:

>  On Tue, Jun 11, 2013 at 10:56:13AM +0800, Mathieu Bridon wrote:
>  > On Mon, 2013-06-10 at 22:36 +1000, Ankur Sinha wrote:
>  > > I was thinking we'd first convert common tags to "standard"
> tags:
>  > > gnome-3 to gnome3, #fedora18 to fedora18 etc. and then get rid of
>  > > unused tags.
>  > I've been doing that quite a lot for the past 2 weeks or so, and was
>  > quite disappointed to see that tags with 0 entries still show up,
>  > and thus encourage people to keep using them. (indeed, I've seen
>  > tags I had cleared being used again for new questions)
>  > Automatically deleting all tags with 0 questions associated seems
>  > like a good thing to do, to ensure higher quality/consistency of
>  > tagging.
>  
>  +1. It's really a mess and this will help. 
Yeah, seems reasonable to me. 

I don't think anything needs to be shut down to run those, but I could
be wrong. Please give it a whirl in staging and if it runs ok there,
enable in production. 

kevin

[Ankur Sinha]

On Tue, 2013-06-11 at 11:15 -0600, Kevin Fenzi wrote:
>  I don't think anything needs to be shut down to run those, but I
> could
>  be wrong.  
Upstream confirmed that the site doesn't need to be shut down

>  Please give it a whirl in staging and if it runs ok there,
>  enable in production.  
I tried it out on stg. Here's what I got. It *seemed* to work, but I'm
uncertain if the COMMIT action was indeed carried out on the db since it
spews these messages. I've seen the deprecation warnings before. Need to
look into them:

[root(a)ask01.stg ~]# python /etc/askbot/sites/ask/config/manage.py
delete_unused_tags
/usr/lib/python2.6/site-packages/django/db/__init__.py:19:
DeprecationWarning: settings.DATABASE_* is deprecated; use
settings.DATABASES instead.
  DeprecationWarning
/usr/lib/python2.6/site-packages/django/db/__init__.py:60:
DeprecationWarning: Short names for ENGINE in database configurations
are deprecated. Prepend default.ENGINE with 'django.db.backends.'
  DeprecationWarning
WARNING!!! You are using a 'locmem' (local memory) caching backend,
which is OK for a low volume site running on a single-process server.
For a multi-process configuration it is neccessary to have a production
cache system, such as redis or memcached.

With local memory caching and multi-process setup you might
intermittently
see outdated content on your site.

Searching for unused tags:
------------------------------------------------------------ 100.00%
Found an unused tag everyone
Deleted.
Traceback (most recent call last):
  File "/etc/askbot/sites/ask/config/manage.py", line 10, in <module>
    execute_manager(settings)
  File
"/usr/lib/python2.6/site-packages/django/core/management/__init__.py",
line 438, in execute_manager
    utility.execute()
  File
"/usr/lib/python2.6/site-packages/django/core/management/__init__.py",
line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
220, in execute
    output = self.handle(*args, **options)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
351, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
222, in inner
    self.__exit__(None, None, None)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
207, in __exit__
    self.exiting(exc_value, self.using)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
302, in exiting
    leave_transaction_management(using=using)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
56, in leave_transaction_management
    connection.leave_transaction_management()
  File
"/usr/lib/python2.6/site-packages/django/db/backends/__init__.py", line
115, in leave_transaction_management
    raise TransactionManagementError("Transaction managed block ended
with "
django.db.transaction.TransactionManagementError: Transaction managed
block ended with pending COMMIT/ROLLBACK
[root(a)ask01.stg ~]# python /etc/askbot/sites/ask/config/manage.py
delete_unused_tags
/usr/lib/python2.6/site-packages/django/db/__init__.py:19:
DeprecationWarning: settings.DATABASE_* is deprecated; use
settings.DATABASES instead.
  DeprecationWarning
/usr/lib/python2.6/site-packages/django/db/__init__.py:60:
DeprecationWarning: Short names for ENGINE in database configurations
are deprecated. Prepend default.ENGINE with 'django.db.backends.'
  DeprecationWarning
WARNING!!! You are using a 'locmem' (local memory) caching backend,
which is OK for a low volume site running on a single-process server.
For a multi-process configuration it is neccessary to have a production
cache system, such as redis or memcached.

With local memory caching and multi-process setup you might
intermittently
see outdated content on your site.

Searching for unused tags:
------------------------------------------------------------ 100.00%
Did not find any.
Traceback (most recent call last):
  File "/etc/askbot/sites/ask/config/manage.py", line 10, in <module>
    execute_manager(settings)
  File
"/usr/lib/python2.6/site-packages/django/core/management/__init__.py",
line 438, in execute_manager
    utility.execute()
  File
"/usr/lib/python2.6/site-packages/django/core/management/__init__.py",
line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
220, in execute
    output = self.handle(*args, **options)
  File
"/usr/lib/python2.6/site-packages/django/core/management/base.py", line
351, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
222, in inner
    self.__exit__(None, None, None)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
207, in __exit__
    self.exiting(exc_value, self.using)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
302, in exiting
    leave_transaction_management(using=using)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line
56, in leave_transaction_management
    connection.leave_transaction_management()
  File
"/usr/lib/python2.6/site-packages/django/db/backends/__init__.py", line
115, in leave_transaction_management
    raise TransactionManagementError("Transaction managed block ended
with "
django.db.transaction.TransactionManagementError: Transaction managed
block ended with pending COMMIT/ROLLBACK
[root(a)ask01.stg ~]#

-- 
Thanks,
Warm regards,
Ankur (FranciscoD)

http://fedoraproject.org/wiki/User:Ankursinha

Join Fedora! Come talk to us!
http://fedoraproject.org/wiki/Fedora_Join_SIG

[Kevin Fenzi]

On Wed, 12 Jun 2013 17:24:45 +1000
Ankur Sinha <sanjay.ankur(a)gmail.com&gt; wrote:

>  On Tue, 2013-06-11 at 11:15 -0600, Kevin Fenzi wrote:
>  > I don't think anything needs to be shut down to run those, but I
>  > could be wrong. 
>  
>  Upstream confirmed that the site doesn't need to be shut down
>  
>  > Please give it a whirl in staging and if it runs ok there,
>  > enable in production. 
>  
>  I tried it out on stg. Here's what I got. It *seemed* to work, but I'm
>  uncertain if the COMMIT action was indeed carried out on the db since
>  it spews these messages. I've seen the deprecation warnings before.
>  Need to look into them: 
...snip...

This looks to me like it failed. Something was malformed. 

I'd ping upstream about it. 

kevin