Koji: ServerOffline

[Jitesh Shah]

Hi,
I've been running a local koji instance at my workplace. It worked
absolutely fine for a fortnight. But, since Monday, I am getting this
error : "ServerOffline: database outage" for any CLI command that I try
to execute.

I initially suspected that it might be because koji cannot connect to
the database. But, I checked that I can login to the database through
command-line.
> > psql

Also, I wrote a small script in python which mimics koji's method of
connecting to the database (with reference
to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine!

Just to make sure, I restarted postgresql service in vain.
Re-initialised the database (Remove /var/lib/pgsql and run "service
postgresql initdb" and all the other steps) to no avail.

I started seeing this problem after a run of the koji-shadow script. It
exited abnormally giving this very error. Since then, I cannot even
access koji-cli.. neither koji-web.

I have been tracing through the koji scripts and I see that the SSL
login succeeds and then the very first rpc call: "getAPIVersion" fails.
Wireshark shows that the koji-hub replies with "Connection: close" to
the getAPIVersion call.

Any idea as to what is going wrong?
Any pointers?

Jitesh

[Oliver Falk]

Hi Jitesh!

Jitesh Shah wrote:
> Hi,
> I've been running a local koji instance at my workplace. It worked
> absolutely fine for a fortnight. But, since Monday, I am getting this
> error : "ServerOffline: database outage" for any CLI command that I try
> to execute.
>
> I initially suspected that it might be because koji cannot connect to
> the database. But, I checked that I can login to the database through
> command-line.
> > > psql
>
> Also, I wrote a small script in python which mimics koji's method of
> connecting to the database (with reference
> to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine!
>
> Just to make sure, I restarted postgresql service in vain.
> Re-initialised the database (Remove /var/lib/pgsql and run "service
> postgresql initdb" and all the other steps) to no avail.
>
> I started seeing this problem after a run of the koji-shadow script. It
> exited abnormally giving this very error. Since then, I cannot even
> access koji-cli.. neither koji-web.
>
> I have been tracing through the koji scripts and I see that the SSL
> login succeeds and then the very first rpc call: "getAPIVersion" fails.
> Wireshark shows that the koji-hub replies with "Connection: close" to
> the getAPIVersion call.
>
> Any idea as to what is going wrong?
> Any pointers?

You have not written if you have tried to restart Apache. koji(hub) has 
no auto-reconnect if it looses connection to the database. So if for 
whatever reason it lost the connection to PostgreSQL, it will never 
reconnect. You have to manually restart Apache!

-of

[Jitesh Shah]

Hi,
Thank you for the quick reply Oliver.

So, this morning I moved to a newer version of koji. Reconfigured
everything (restarted apache after re-configuring koji-hub) and quite
surprisingly, I still get the same error!

So, if reinitialising both, postgresql and koji, doesn't do the trick, I
wonder what might have gone wrong.
(Just as a matter of curiosity, I even checked with ipcs that there are
no dangling shared memory segments. There aren't any.)

Jitesh

On Thu, 2009-01-22 at 08:16 +0100, Oliver Falk wrote:
> Hi Jitesh!
>
> Jitesh Shah wrote:
> > Hi,
> > I've been running a local koji instance at my workplace. It worked
> > absolutely fine for a fortnight. But, since Monday, I am getting this
> > error : "ServerOffline: database outage" for any CLI command that I try
> > to execute.
> >
> > I initially suspected that it might be because koji cannot connect to
> > the database. But, I checked that I can login to the database through
> > command-line.
> > > > psql
> >
> > Also, I wrote a small script in python which mimics koji's method of
> > connecting to the database (with reference
> > to /usr/lib/python2.5/site-packages/koji/db.py) and it works fine!
> >
> > Just to make sure, I restarted postgresql service in vain.
> > Re-initialised the database (Remove /var/lib/pgsql and run "service
> > postgresql initdb" and all the other steps) to no avail.
> >
> > I started seeing this problem after a run of the koji-shadow script. It
> > exited abnormally giving this very error. Since then, I cannot even
> > access koji-cli.. neither koji-web.
> >
> > I have been tracing through the koji scripts and I see that the SSL
> > login succeeds and then the very first rpc call: "getAPIVersion" fails.
> > Wireshark shows that the koji-hub replies with "Connection: close" to
> > the getAPIVersion call.
> >
> > Any idea as to what is going wrong?
> > Any pointers?
>
> You have not written if you have tried to restart Apache. koji(hub) has 
> no auto-reconnect if it looses connection to the database. So if for 
> whatever reason it lost the connection to PostgreSQL, it will never 
> reconnect. You have to manually restart Apache!
>
> -of
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Oliver Falk]

Jitesh Shah wrote:
> So, this morning I moved to a newer version of koji. Reconfigured
> everything (restarted apache after re-configuring koji-hub) and quite
> surprisingly, I still get the same error!
>
> So, if reinitialising both, postgresql and koji, doesn't do the trick, I
> wonder what might have gone wrong.
> (Just as a matter of curiosity, I even checked with ipcs that there are
> no dangling shared memory segments. There aren't any.)

The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess?
Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is 
also OK?

And you can connect via command line? psql -U <DBUser> -h <DBHost> <DBName>?

The error is quite clear. kojihub is not able to connect to the 
database, so something is wrong with the config... :-(

-of

[Toshio Kuratomi]

Oliver Falk wrote:
> The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess?
> Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is
> also OK?
>
> And you can connect via command line? psql -U <DBUser> -h <DBHost>
> <DBName>?

... and the psql command is run on the same machine as koji?

[Oliver Falk]

Toshio Kuratomi wrote:
> Oliver Falk wrote:
> > The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess?
> > Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is
> > also OK?
> >
> > And you can connect via command line? psql -U <DBUser> -h <DBHost>
> > <DBName>?
>
> ... and the psql command is run on the same machine as koji?

Well I should have mentioned this, thx Toshio for pointing this out!

-of

[Jitesh Shah]

Hi,
So, I had koji running and building packages for me for nearly a
fortnight. And it worked like a charm!

I double-checked all the configuration and it is intact. There is the
and entry in pg_hba.conf that trusts koji.

I run psql command from the "koji" user on the same machine where
koji-hub is running. So, psql uses the default values for DBName and
DBUser. (which are "koji" and "koji" respectively) (And I don't use the
host option). Also, I've checked that DBName and DBUser configuration
are OK in the kojihub.conf

On a different note, how does one debug a script which uses rpc calls
and the error is on the server side? i.e. the debugger will run on the
caller side, so, how do we debug the script that is executed on the
server? (Except inserting write-to-file commands in the server script)
Also, does koji maintain a logfile?... I wasn't able find it anywhere.

Jitesh

On Thu, 2009-01-22 at 08:45 +0100, Oliver Falk wrote:
> Jitesh Shah wrote:
> > So, this morning I moved to a newer version of koji. Reconfigured
> > everything (restarted apache after re-configuring koji-hub) and quite
> > surprisingly, I still get the same error!
> >
> > So, if reinitialising both, postgresql and koji, doesn't do the trick, I
> > wonder what might have gone wrong.
> > (Just as a matter of curiosity, I even checked with ipcs that there are
> > no dangling shared memory segments. There aren't any.)
>
> The settings in /etc/httpd/conf.d/kojihub.conf are correct I guess?
> Esp. DBName, DBUser, DBHost? And your /var/lib/pgsql/data/pg_hba.conf is 
> also OK?
>
> And you can connect via command line? psql -U <DBUser> -h <DBHost> <DBName>?
>
> The error is quite clear. kojihub is not able to connect to the 
> database, so something is wrong with the config... :-(
>
> -of
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Toshio Kuratomi]

Jitesh Shah wrote:
> Hi,
> So, I had koji running and building packages for me for nearly a
> fortnight. And it worked like a charm!
>
> I double-checked all the configuration and it is intact. There is the
> and entry in pg_hba.conf that trusts koji.
>
> I run psql command from the "koji" user on the same machine where
> koji-hub is running. So, psql uses the default values for DBName and
> DBUser. (which are "koji" and "koji" respectively) (And I don't use the
> host option). Also, I've checked that DBName and DBUser configuration
> are OK in the kojihub.conf

It sounds like you're running the koji db on the same machine as the
kojihub?  Just to be sure you're testing the same thing with psql as
koji is going to run, try  psql with the -h HOST option.  The reason is
that psql without the -h will try to use unix sockets to connect on the
local machine.  koji, though, is going to use a tcp connection, not a
socket.

-Toshio

[Jitesh Shah]

Hi,
Yes, I can successfully connect to the psql database using tcp sockets
too. Here is the snippet of the interaction in the terminal.

[koji@linux-dev ~]$ psql -U koji -h localhost -d koji
Welcome to psql 8.2.11, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       ? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

koji=>

I can view all the tables with "\d". So, it is connected all-right. Btw,
I also checked in the "users" "permissions" and "user_perms" tables to
verify that "koji" user is indeed the "admin" and has all the required
privileges.

Jitesh

On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote:
> Jitesh Shah wrote:
> > Hi,
> > So, I had koji running and building packages for me for nearly a
> > fortnight. And it worked like a charm!
> >
> > I double-checked all the configuration and it is intact. There is the
> > and entry in pg_hba.conf that trusts koji.
> >
> > I run psql command from the "koji" user on the same machine where
> > koji-hub is running. So, psql uses the default values for DBName and
> > DBUser. (which are "koji" and "koji" respectively) (And I don't use the
> > host option). Also, I've checked that DBName and DBUser configuration
> > are OK in the kojihub.conf
>
> It sounds like you're running the koji db on the same machine as the
> kojihub?  Just to be sure you're testing the same thing with psql as
> koji is going to run, try  psql with the -h HOST option.  The reason is
> that psql without the -h will try to use unix sockets to connect on the
> local machine.  koji, though, is going to use a tcp connection, not a
> socket.
>
> -Toshio
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Oliver Falk]

You could do the following on your kojihub - I guess it's only running 
koji!? If not that's bad :-)

# service httpd stop
# strace -f httpd -X

You may want to write it to some strace.log (add 2>&1 > strace.log).

Try to analyse this. You should see where it tries to connect to.

-of

Jitesh Shah wrote:
> Hi,
> Yes, I can successfully connect to the psql database using tcp sockets
> too. Here is the snippet of the interaction in the terminal.
>
> [koji@linux-dev ~]$ psql -U koji -h localhost -d koji
> Welcome to psql 8.2.11, the PostgreSQL interactive terminal.
>
> Type:  \copyright for distribution terms
>        \h for help with SQL commands
>        ? for help with psql commands
>        \g or terminate with semicolon to execute query
>        \q to quit
>
> koji=>
>
> I can view all the tables with "\d". So, it is connected all-right. Btw,
> I also checked in the "users" "permissions" and "user_perms" tables to
> verify that "koji" user is indeed the "admin" and has all the required
> privileges.
>
> Jitesh
>
> On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote:
> > Jitesh Shah wrote:
> > > Hi,
> > > So, I had koji running and building packages for me for nearly a
> > > fortnight. And it worked like a charm!
> > >
> > > I double-checked all the configuration and it is intact. There is the
> > > and entry in pg_hba.conf that trusts koji.
> > >
> > > I run psql command from the "koji" user on the same machine where
> > > koji-hub is running. So, psql uses the default values for DBName and
> > > DBUser. (which are "koji" and "koji" respectively) (And I don't use the
> > > host option). Also, I've checked that DBName and DBUser configuration
> > > are OK in the kojihub.conf
> >
> > It sounds like you're running the koji db on the same machine as the
> > kojihub?  Just to be sure you're testing the same thing with psql as
> > koji is going to run, try  psql with the -h HOST option.  The reason is
> > that psql without the -h will try to use unix sockets to connect on the
> > local machine.  koji, though, is going to use a tcp connection, not a
> > socket.
> >
> > -Toshio
> >
> > --
> > Fedora-buildsys-list mailing list
> > [e-mail]
> > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Jitesh Shah]

Hi!
It finally worked!!! Rejoice!

So, while stracing httpd, as suggested by Oliver, I realised that koji
was _NOT_ using localhost to connect to psql because the DBHost
parameter in the koji configuration had been tampered with and changed
to something else. And obviously the request was then denied.

Anyway, Thanks to all of you and specially Oliver for pointing me in the
right direction!

Jitesh

On Thu, 2009-01-22 at 13:28 +0100, Oliver Falk wrote:
> You could do the following on your kojihub - I guess it's only running 
> koji!? If not that's bad :-)
>
> # service httpd stop
> # strace -f httpd -X
>
> You may want to write it to some strace.log (add 2>&1 > strace.log).
>
> Try to analyse this. You should see where it tries to connect to.
>
> -of
>
> Jitesh Shah wrote:
> > Hi,
> > Yes, I can successfully connect to the psql database using tcp sockets
> > too. Here is the snippet of the interaction in the terminal.
> >
> > [koji@linux-dev ~]$ psql -U koji -h localhost -d koji
> > Welcome to psql 8.2.11, the PostgreSQL interactive terminal.
> >
> > Type:  \copyright for distribution terms
> >        \h for help with SQL commands
> >        ? for help with psql commands
> >        \g or terminate with semicolon to execute query
> >        \q to quit
> >
> > koji=>
> >
> > I can view all the tables with "\d". So, it is connected all-right. Btw,
> > I also checked in the "users" "permissions" and "user_perms" tables to
> > verify that "koji" user is indeed the "admin" and has all the required
> > privileges.
> >
> > Jitesh
> >
> > On Thu, 2009-01-22 at 02:18 -0800, Toshio Kuratomi wrote:
> > > Jitesh Shah wrote:
> > > > Hi,
> > > > So, I had koji running and building packages for me for nearly a
> > > > fortnight. And it worked like a charm!
> > > >
> > > > I double-checked all the configuration and it is intact. There is the
> > > > and entry in pg_hba.conf that trusts koji.
> > > >
> > > > I run psql command from the "koji" user on the same machine where
> > > > koji-hub is running. So, psql uses the default values for DBName and
> > > > DBUser. (which are "koji" and "koji" respectively) (And I don't use the
> > > > host option). Also, I've checked that DBName and DBUser configuration
> > > > are OK in the kojihub.conf
> > >
> > > It sounds like you're running the koji db on the same machine as the
> > > kojihub?  Just to be sure you're testing the same thing with psql as
> > > koji is going to run, try  psql with the -h HOST option.  The reason is
> > > that psql without the -h will try to use unix sockets to connect on the
> > > local machine.  koji, though, is going to use a tcp connection, not a
> > > socket.
> > >
> > > -Toshio
> > >
> > > --
> > > Fedora-buildsys-list mailing list
> > > [e-mail]
> > > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
> >
> > --
> > Fedora-buildsys-list mailing list
> > [e-mail]
> > https://www.redhat.com/mailman/listinfo/fedora-buildsys-list
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Oliver Falk]

Hi!

Jitesh Shah wrote:
> It finally worked!!! Rejoice!

Great.
> So, while stracing httpd, as suggested by Oliver, I realised that koji
> was _NOT_ using localhost to connect to psql because the DBHost
> parameter in the koji configuration had been tampered with and changed
> to something else. And obviously the request was then denied.

As I always say. strace is your friend. :-P
> Anyway, Thanks to all of you and specially Oliver for pointing me in the
> right direction!

Crazy me. I'm not even one of the koji developers... :-) Yay, yay. It's 
all about community, isn't it? 8-]

-of

[Mike McLean]

Jitesh Shah wrote:
> On a different note, how does one debug a script which uses rpc calls
> and the error is on the server side? i.e. the debugger will run on the
> caller side, so, how do we debug the script that is executed on the
> server? (Except inserting write-to-file commands in the server script)
> Also, does koji maintain a logfile?... I wasn't able find it anywhere.

kojihub logs to stderr, which under apache goes to 
/var/log/httpd/error_log (or ssl_error_log). If you don't see anything 
helpful, try setting KojiDebug to "on" in the config and restarting httpd.

Which version of koji are you running?

[Oliver Falk]

Mike McLean wrote:
> Jitesh Shah wrote:
> > On a different note, how does one debug a script which uses rpc calls
> > and the error is on the server side? i.e. the debugger will run on the
> > caller side, so, how do we debug the script that is executed on the
> > server? (Except inserting write-to-file commands in the server script)
> > Also, does koji maintain a logfile?... I wasn't able find it anywhere.
>
> kojihub logs to stderr, which under apache goes to 
> /var/log/httpd/error_log (or ssl_error_log). If you don't see anything 
> helpful, try setting KojiDebug to "on" in the config and restarting httpd.

I wonder if it would log this error in a way you know what's going on :-)

Jitesh, can you check the logs as suggested by Mike, although it's fixed 
now, I'd like to know if you see the error there...

[ ... ]

-of

[Jitesh Shah]

On Thu, 2009-01-22 at 17:17 +0100, Oliver Falk wrote:
> Mike McLean wrote:
> I wonder if it would log this error in a way you know what's going on :-)
>
> Jitesh, can you check the logs as suggested by Mike, although it's fixed 
> now, I'd like to know if you see the error there...

I checked the logs. There is just a python traceback and the actual
error "ServerOffline: Database Outage' (This is with KojiDebug off)

With KojiDebug turned on, it additionally shows the queries executed and
the time taken for each.

Nothing that would point to an incorrect configuration or a failure in
database connection.

Jitesh
> [ ... ]
>
> -of
>
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Oliver Falk]

Jitesh Shah wrote:
> On Thu, 2009-01-22 at 17:17 +0100, Oliver Falk wrote:
> > Mike McLean wrote:
> > I wonder if it would log this error in a way you know what's going on :-)
> >
> > Jitesh, can you check the logs as suggested by Mike, although it's fixed
> > now, I'd like to know if you see the error there...
>
> I checked the logs. There is just a python traceback and the actual
> error "ServerOffline: Database Outage' (This is with KojiDebug off)
>
> With KojiDebug turned on, it additionally shows the queries executed and
> the time taken for each.
>
> Nothing that would point to an incorrect configuration or a failure in
> database connection.

Fine. Then it wasn't wrong to give you the strace hint :-)

-of

[Mike McLean]

Jitesh Shah wrote:
> I checked the logs. There is just a python traceback and the actual
> error "ServerOffline: Database Outage' (This is with KojiDebug off)

'just' a python traceback? Often these are the key to the mystery.

[Jitesh Shah]

Hi,

On Thu, 2009-01-22 at 10:53 -0500, Mike McLean wrote:
> Jitesh Shah wrote:
> Which version of koji are you running?

I am using 1.2.6-1
> --
> Fedora-buildsys-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/fedora-buildsys-list

[Mike McLean]

Oliver Falk wrote:
> You have not written if you have tried to restart Apache. koji(hub) has 
> no auto-reconnect if it looses connection to the database. So if for 
> whatever reason it lost the connection to PostgreSQL, it will never 
> reconnect. You have to manually restart Apache!

Actually, it does reconnect now.

[Oliver Falk]

Mike McLean wrote:
> Oliver Falk wrote:
> > You have not written if you have tried to restart Apache. koji(hub) 
> > has no auto-reconnect if it looses connection to the database. So if 
> > for whatever reason it lost the connection to PostgreSQL, it will 
> > never reconnect. You have to manually restart Apache!
>
> Actually, it does reconnect now.

Since which version?

-of

[Mike McLean]

Oliver Falk wrote:
> Mike McLean wrote:
> > Actually, it does reconnect now.
>
> Since which version?

The change is in commit 58b7313d. Should be in 1.2.6
https://fedorahosted.org/koji/changeset/58b7313d0af0e0ac1fe804cd9f247486b313...

[Oliver Falk]

Mike McLean schrieb:
> Oliver Falk wrote:
> > Mike McLean wrote:
> > > Actually, it does reconnect now.
> >
> > Since which version?
>
> The change is in commit 58b7313d. Should be in 1.2.6
> https://fedorahosted.org/koji/changeset/58b7313d0af0e0ac1fe804cd9f247486b313...

Well, thx. Good to know this is 'fixed'.

-of