jboss resource: no alert clearing

[Attila Heidrich]

Please take a look at the screenshots here:
https://www.dropbox.com/sh/heyowe3rc145ivd/_tlWMpMG3F

The resource is available, as it is shown in the timeline, but still there
are live alerts.
any idea, what is wrong? This is exactly the same as two days before, even
the TOD is the same.

I have found something at the given time in the Cassandra log:

INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 MessagingService.java
(line 860) 1 READ messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 StatusLogger.java (line
53) Pool Name                    Active   Pending      Completed   Blocked
All Time Blocked
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java (line
68) ReadStage                         0         0       23386785
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java (line
68) RequestResponseStage              0         0       43148496
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) ReadRepairStage                   0         0        1722951
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) MutationStage                     0         0       76756647
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) ReplicateOnWriteStage             0         0              0
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java (line
68) GossipStage                       0         0       11364876
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) AntiEntropyStage                  0         0          69120
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) MigrationStage                    0         0              0
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) MemtablePostFlusher               0         0          84294
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java (line
68) FlushWriter                       0         0           1277
0                 3
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) MiscStage                         0         0            201
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) commitlog_archiver                0         0              0
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java (line
68) InternalResponseStage             0         0              0
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
68) AntiEntropySessions               0         0           2048
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
68) HintedHandoff                     0         0              5
0                 0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
78) CompactionManager                 0         0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java (line
90) MessagingService                n/a       0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
100) Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
101) KeyCache                   38797256
38797312                      all
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
107) RowCache                          0
0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
114) ColumnFamily                Memtable ops,data
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
117) system.local                              0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java (line
117) system.peers                              0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.batchlog                           0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.NodeIdInfo                         0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.LocationInfo                       0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.Schema                             0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.Migrations                         0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java (line
117) system.schema_keyspaces                   0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.schema_columns                     0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.schema_columnfamilies                 0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.IndexInfo                          0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.range_xfers                        0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.peer_events                        0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java (line
117) system.hints                              0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) system.HintsColumnFamily                  0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.twenty_four_hour_metrics      139188,2097152
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.raw_metrics               870881,14680064
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.six_hour_metrics          487836,13631488
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.metrics_index              660267,4194304
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.schema_version                        0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java (line
117) rhq.one_hour_metrics           163941,5242880
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.users                         0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.credentials                   0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_auth.permissions                   0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_traces.sessions                    0,0
 INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java (line
117) system_traces.events                      0,0

Similar happens sometimes, but doesn't look to be stronglyrelated to my
problem, since there is no such info from 28-01.

/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-02 04:09:21,020 MessagingService.java (line 860) 3 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-06 04:04:16,964 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-07 04:11:18,930 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-14 04:11:49,388 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-14 04:15:19,999 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-21 04:07:32,958 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-23 04:06:56,011 MessagingService.java (line 860) 3 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-23 04:10:53,667 MessagingService.java (line 860) 2 READ messages
dropped in last 5000ms
/opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1]
2014-01-30 04:10:19,381 MessagingService.java (line 860) 1 READ messages
dropped in last 5000ms

Any idea? Last time I had to restart the agent. Then I have received the
clearing message two times.

Regards,

Attila

[Jay Shaughnessy]

Are you saying you expected a "Goes UP" alert and you didn't get one?

On 1/30/2014 4:17 AM, Attila Heidrich wrote:
> Please take a look at the screenshots here: 
> https://www.dropbox.com/sh/heyowe3rc145ivd/_tlWMpMG3F
>
> The resource is available, as it is shown in the timeline, but still 
> there are live alerts.
> any idea, what is wrong? This is exactly the same as two days before, 
> even the TOD is the same.
>
> I have found something at the given time in the Cassandra log:
>
> INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 MessagingService.java 
> (line 860) 1 READ messages dropped in last 5000ms
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,381 StatusLogger.java 
> (line 53) Pool Name Active   Pending      Completed   Blocked  All 
> Time Blocked
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java 
> (line 68) ReadStage                         0         0 
> 23386785         0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,382 StatusLogger.java 
> (line 68) RequestResponseStage              0         0 
> 43148496         0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java 
> (line 68) ReadRepairStage                   0         0 
> 1722951         0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java 
> (line 68) MutationStage                     0         0 
> 76756647         0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java 
> (line 68) ReplicateOnWriteStage             0         0 0         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,383 StatusLogger.java 
> (line 68) GossipStage                       0         0 
> 11364876         0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java 
> (line 68) AntiEntropyStage                  0         0 69120         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java 
> (line 68) MigrationStage                    0         0 0         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java 
> (line 68) MemtablePostFlusher               0         0 84294         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,384 StatusLogger.java 
> (line 68) FlushWriter                       0         0 1277         
> 0                 3
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java 
> (line 68) MiscStage                         0         0 201         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java 
> (line 68) commitlog_archiver                0         0 0         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,385 StatusLogger.java 
> (line 68) InternalResponseStage             0         0 0         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java 
> (line 68) AntiEntropySessions               0         0 2048         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java 
> (line 68) HintedHandoff                     0         0 5         
> 0                 0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java 
> (line 78) CompactionManager                 0         0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,386 StatusLogger.java 
> (line 90) MessagingService n/a       0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 100) Cache Type Size                 Capacity KeysToSave Provider
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 101) KeyCache 38797256                 
> 38797312                      all
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 107) RowCache                          0 0                      
> all org.apache.cassandra.cache.SerializingCacheProvider
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 114) ColumnFamily Memtable ops,data
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 117) system.local                              0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,387 StatusLogger.java 
> (line 117) system.peers                              0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.batchlog                           0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.NodeIdInfo                         0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.LocationInfo                       0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.Schema                             0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.Migrations                         0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,388 StatusLogger.java 
> (line 117) system.schema_keyspaces                   0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.schema_columns                     0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.schema_columnfamilies                 0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.IndexInfo                          0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.range_xfers                        0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.peer_events                        0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,389 StatusLogger.java 
> (line 117) system.hints                              0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) system.HintsColumnFamily                  0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.twenty_four_hour_metrics      139188,2097152
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.raw_metrics 870881,14680064
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.six_hour_metrics 487836,13631488
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.metrics_index 660267,4194304
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.schema_version                        0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,390 StatusLogger.java 
> (line 117) rhq.one_hour_metrics 163941,5242880
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java 
> (line 117) system_auth.users                         0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java 
> (line 117) system_auth.credentials                   0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java 
> (line 117) system_auth.permissions                   0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java 
> (line 117) system_traces.sessions                    0,0
>  INFO [ScheduledTasks:1] 2014-01-30 04:10:19,391 StatusLogger.java 
> (line 117) system_traces.events                      0,0
>
> Similar happens sometimes, but doesn't look to be stronglyrelated to 
> my problem, since there is no such info from 28-01.
>
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-02 04:09:21,020 MessagingService.java (line 860) 3 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-06 04:04:16,964 MessagingService.java (line 860) 1 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-07 04:11:18,930 MessagingService.java (line 860) 2 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-14 04:11:49,388 MessagingService.java (line 860) 2 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-14 04:15:19,999 MessagingService.java (line 860) 1 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-21 04:07:32,958 MessagingService.java (line 860) 1 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-23 04:06:56,011 MessagingService.java (line 860) 3 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-23 04:10:53,667 MessagingService.java (line 860) 2 READ 
> messages dropped in last 5000ms
> /opt/rhq-server-4.9.0/logs/rhq-storage.log: INFO [ScheduledTasks:1] 
> 2014-01-30 04:10:19,381 MessagingService.java (line 860) 1 READ 
> messages dropped in last 5000ms
>
> Any idea? Last time I had to restart the agent. Then I have received 
> the clearing message two times.
>
> Regards,
>
> Attila
>
> rhq-users mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/rhq-users