Outdated clamav database in Taskotron

[Petr Pisar]

I received this test failure
https://taskotron.fedoraproject.org/artifacts/all/ad295f84-7690-11e7-a988-5254008e42f6/task_output/rpmgrill.json:

{
         "module" : "VirusCheck",
         "order" : 2,
         "results" : [
            {
               "arch" : "src",
               "code" : "ClamAV",
               "context" : {
                  "path" : "/libwbxml-0.11.5.tar.bz2"
               },
               "diag" : "ClamAV <b>Win.Trojan.Ramnit-5550</b> subtest triggered (ClamAV 0.99.2/21723/Mon Jun 13 13:53:00 2016)",
               "subpackage" : "libwbxml"
            }
         ],
         "run_time" : 9,
         "status" : "completed"
      },

I checked it locally in Fedora 27 with the same result. The tarball contains
win32/expat/libexpat.dll that triggers the antivirus.

But look at the clamav database version (21723/Mon Jun 13 13:53:00 2016). It's
more than a year old. After updating the database (you need to install
clamav-update package and wait for a cronjob or run /usr/bin/freshclam as root
manually), the hit went away. It's a false positive caused by Tasktron not
updating the virus database.

Please update the virus database regularly.

-- Petr

[Kamil Paral]

On Tue, Aug 1, 2017 at 11:18 AM, Petr Pisar  wrote:
> I received this test failure
> https://taskotron.fedoraproject.org/artifacts/
> all/ad295f84-7690-11e7-a988-5254008e42f6/task_output/rpmgrill.json:
>
> {
>          "module" : "VirusCheck",
>          "order" : 2,
>          "results" : [
>             {
>                "arch" : "src",
>                "code" : "ClamAV",
>                "context" : {
>                   "path" : "/libwbxml-0.11.5.tar.bz2"
>                },
>                "diag" : "ClamAV <b>Win.Trojan.Ramnit-5550</b> subtest
> triggered (ClamAV 0.99.2/21723/Mon Jun 13 13:53:00 2016)",
>                "subpackage" : "libwbxml"
>             }
>          ],
>          "run_time" : 9,
>          "status" : "completed"
>       },
>
> I checked it locally in Fedora 27 with the same result. The tarball
> contains
> win32/expat/libexpat.dll that triggers the antivirus.
>
> But look at the clamav database version (21723/Mon Jun 13 13:53:00 2016).
> It's
> more than a year old. After updating the database (you need to install
> clamav-update package and wait for a cronjob or run /usr/bin/freshclam as
> root
> manually), the hit went away. It's a false positive caused by Tasktron not
> updating the virus database.
>
> Please update the virus database regularly.
>
> -- Petr

Hi Petr,

thanks for the report. In the task, we just install rpmgrill and run it. If
rpmgrill reports outdated clamav results, it seems that's something that
should be fixed in rpmgrill itself (it could depend on clamav-update and
update the virus db before running the virus check). Can you please report
a bug against rpmgrill and post the link here? (We could do it ourselves
before running rpmgrill, but it seems cleaner to me if rpmgrill itself
performs this, because that's the tool responsible for running the virus
check - we're just a wrapper around it that schedules in on every koji
build. Furthermore, having it in rpmgrill will fix this for all users of
rpmgrill, regardless of execution). Thanks a lot.

Kamil

[Petr Pisar]

On Tue, Aug 01, 2017 at 11:59:41AM +0200, Kamil Paral wrote:
> thanks for the report. In the task, we just install rpmgrill and run it. If
> rpmgrill reports outdated clamav results, it seems that's something that
> should be fixed in rpmgrill itself (it could depend on clamav-update and
> update the virus db before running the virus check). Can you please report
> a bug against rpmgrill and post the link here?

https://bugzilla.redhat.com/show_bug.cgi?id=1477130

-- Petr

[Róman Joost]

Dear Petr,

On Tue, Aug 01, 2017 at 01:05:34PM +0200, Petr Pisar wrote:
> On Tue, Aug 01, 2017 at 11:59:41AM +0200, Kamil Paral wrote:
> > thanks for the report. In the task, we just install rpmgrill and run it. If
> > rpmgrill reports outdated clamav results, it seems that's something that
> > should be fixed in rpmgrill itself (it could depend on clamav-update and
> > update the virus db before running the virus check). Can you please report
> > a bug against rpmgrill and post the link here?
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1477130

Many thanks for the report. I haven't looked into the specifics, but I'm
not sure what we envision rpmgrill should be doing here. Should it run a
freshclam every invokation of rpmgrill? From what I see, that can take
quite a bit of time.

Maybe what could be done tho is a systemd timer installed with the
package which runs freshclam every now and than?

Kind Regards,

[Dan Callaghan]

Excerpts from Róman Joost's message of 2017-08-02 10:18 +10:00:
> Dear Petr,
>
> On Tue, Aug 01, 2017 at 01:05:34PM +0200, Petr Pisar wrote:
> > On Tue, Aug 01, 2017 at 11:59:41AM +0200, Kamil Paral wrote:
> > > thanks for the report. In the task, we just install rpmgrill and run it. If
> > > rpmgrill reports outdated clamav results, it seems that's something that
> > > should be fixed in rpmgrill itself (it could depend on clamav-update and
> > > update the virus db before running the virus check). Can you please report
> > > a bug against rpmgrill and post the link here?
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1477130
>
> Many thanks for the report. I haven't looked into the specifics, but I'm
> not sure what we envision rpmgrill should be doing here. Should it run a
> freshclam every invokation of rpmgrill? From what I see, that can take
> quite a bit of time.
>
> Maybe what could be done tho is a systemd timer installed with the
> package which runs freshclam every now and than?

This might not work well if rpmgrill is invoked as part of some system 
which creates a fresh VM from scratch and then deletes it shortly after 
(think single-use slaves with Jenkins OpenStack Cloud plugin, for 
example). The freshclam timer will likely never trigger before rpmgrill 
is run.

One possible thing which might help is if rpmgrill could warn or even 
fail, if it detects that it's being run with "outdated" ClamAV 
definitions. Not sure how old you want to consider "outdated", or if 
there is an easy way to check it... At worst I guess something like, if 
modtime of ClamAV definitions is more than 4 weeks in the past give an 
error? Do we know how frequently the definitions are updated?

The other thing is that this idea of "download some data from the 
internet in order to make this package work" is not a good approach. It 
breaks in exactly the scenario I mentioned above, where a freshly 
installed copy of the package is not actually usable. The pciids and 
usbids database used to be like this too (shipping some old version of 
the data, plus a cron job to pull down updates from the internet) but 
nowadays we have the hwdata package which just gets updated with the 
latest definitions once per month. This is a much nicer solution because 
it means you can install a machine using only Fedora packages (or 
a freshly built disk image) and it already has the data it needs, 
without then going back to some random server on the internet.

So maybe the ClamAV definitions should be treated similarly? In 
a separate package which gets updated on a regular interval to pull in 
the latest data?

[Kamil Paral]

On Wed, Aug 2, 2017 at 2:44 AM, Dan Callaghan  wrote:
> Excerpts from Róman Joost's message of 2017-08-02 10:18 +10:00:
> > Dear Petr,
> >
> > On Tue, Aug 01, 2017 at 01:05:34PM +0200, Petr Pisar wrote:
> > > On Tue, Aug 01, 2017 at 11:59:41AM +0200, Kamil Paral wrote:
> > > > thanks for the report. In the task, we just install rpmgrill and run
>
> it. If
> > > > rpmgrill reports outdated clamav results, it seems that's something
>
> that
> > > > should be fixed in rpmgrill itself (it could depend on clamav-update
>
> and
> > > > update the virus db before running the virus check). Can you please
>
> report
> > > > a bug against rpmgrill and post the link here?
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1477130
> >
> > Many thanks for the report. I haven't looked into the specifics, but I'm
> > not sure what we envision rpmgrill should be doing here. Should it run a
> > freshclam every invokation of rpmgrill? From what I see, that can take
> > quite a bit of time.

That was my initial idea, yes. But now that I tried it, I see two problems:
a) freshclam needs to run as root, therefore rpmgrill executed under a
standard user can't update it anyway. It could warn about it, and trigger
the update if running under root (taskotron case), but it's not a generic
solution which makes sure the results are up-to-date.
b) The database refresh is SLOW. I expected just a few seconds. But here's
my experience:
https://paste.fedoraproject.org/paste/3oGdfMpOP84~rhQm34AZvw/raw
Three minutes full of timing out. This can probably be much longer, if
servers are in even a worse condition some day.
> > Maybe what could be done tho is a systemd timer installed with the
> > package which runs freshclam every now and than?
>
> This might not work well if rpmgrill is invoked as part of some system
> which creates a fresh VM from scratch and then deletes it shortly after
> (think single-use slaves with Jenkins OpenStack Cloud plugin, for
> example). The freshclam timer will likely never trigger before rpmgrill
> is run.

That's Taskotron case, yes. After looking more into this, I guess I
wouldn't object updating clamav database before running rpmgrill. The
problem I have with it is how slow it is. We run rpmgrill on every new koji
builds, that means very frequently. Each run is performed on a clean
machine, meaning each rpmgrill execution takes 3+ minutes longer just
because of clamav servers being horrible. I'd definitely add a timeout and
kill the process if it did not finish in 5 minutes or so, but even the
usual e.g. 3 minutes of extra execution time (mostly idle time) is
something I don't like.
> One possible thing which might help is if rpmgrill could warn or even
> fail, if it detects that it's being run with "outdated" ClamAV
> definitions. Not sure how old you want to consider "outdated", or if
> there is an easy way to check it... At worst I guess something like, if
> modtime of ClamAV definitions is more than 4 weeks in the past give an
> error? Do we know how frequently the definitions are updated?
>
> The other thing is that this idea of "download some data from the
> internet in order to make this package work" is not a good approach. It
> breaks in exactly the scenario I mentioned above, where a freshly
> installed copy of the package is not actually usable. The pciids and
> usbids database used to be like this too (shipping some old version of
> the data, plus a cron job to pull down updates from the internet) but
> nowadays we have the hwdata package which just gets updated with the
> latest definitions once per month. This is a much nicer solution because
> it means you can install a machine using only Fedora packages (or
> a freshly built disk image) and it already has the data it needs,
> without then going back to some random server on the internet.

Very much agreed.
> So maybe the ClamAV definitions should be treated similarly? In
> a separate package which gets updated on a regular interval to pull in
> the latest data?

That would be the best solution here, yes. Could someone please file an RFE
against clamav?

[Róman Joost]

On Wed, Aug 02, 2017 at 04:53:20PM +0200, Kamil Paral wrote:
> On Wed, Aug 2, 2017 at 2:44 AM, Dan Callaghan  wrote:
> [...]
> > The other thing is that this idea of "download some data from the
> > internet in order to make this package work" is not a good approach. It
> > breaks in exactly the scenario I mentioned above, where a freshly
> > installed copy of the package is not actually usable. The pciids and
> > usbids database used to be like this too (shipping some old version of
> > the data, plus a cron job to pull down updates from the internet) but
> > nowadays we have the hwdata package which just gets updated with the
> > latest definitions once per month. This is a much nicer solution because
> > it means you can install a machine using only Fedora packages (or
> > a freshly built disk image) and it already has the data it needs,
> > without then going back to some random server on the internet.
>
> Very much agreed.
> > So maybe the ClamAV definitions should be treated similarly? In
> > a separate package which gets updated on a regular interval to pull in
> > the latest data?
>
> That would be the best solution here, yes. Could someone please file an RFE
> against clamav?

Done:

https://bugzilla.redhat.com/show_bug.cgi?id=1477777

PS: I like the bug number :))

Kind Regards,

[Kamil Paral]

> > > So maybe the ClamAV definitions should be treated similarly? In
> > > a separate package which gets updated on a regular interval to pull in
> > > the latest data?
> >
> > That would be the best solution here, yes. Could someone please file an
>
> RFE
> > against clamav?
>
> Done:
> https://bugzilla.redhat.com/show_bug.cgi?id=1477777

The clamav maintainer doesn't seem to have the time needed to provide a
regularly updated clamav-data package. So today I tried to update the
rpmgrill task with running freshclam before running rpmgrill, using
clamav.upjs.sk mirror. That mirror is very fast when I try it (as opposed
to the default mirror). However, when running it in our infra, I hit
"WARNING: Mirror 158.197.16.70 is not synchronized." errors with
clamav.upjs.sk mirror [1] and extremely long times with the default mirror
(my patience ran out after 5 minutes).

So, sorry, I give up. Unless we can convince someone to maintain a
regularly updated clamav-data package, or provide a reliable and fast
clamav mirror (that is willing to be hit very often from our infra), I'm
afraid we can't offer an updated clamav database for our tests.

[1] https://paste.fedoraproject.org/paste/hqewIGsDK8XFn9O6B0bPPQ/raw