Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 785ca18f2b2963555ed8e39e17676befcf8c2406 https://github.com/rhq-project/rhq/commit/785ca18f2b2963555ed8e39e17676befcf... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message: ----------- [BZ 185375] speed up data migration step
Writes are now done in batches. Reads are done in parallel and concurrently in batches.
Commit: 59c745701735920756b7579d9a11a9773bc9ac6f https://github.com/rhq-project/rhq/commit/59c745701735920756b7579d9a11a9773b... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/DateUtils.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/Replace412Index.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceRHQ411Index.java M modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/ReplaceIndexTest.java
Log Message: ----------- [BZ 185375] refactor date functions into util class
Commit: 4d07cd682e5084f81e4171e1265b9f0c22488c38 https://github.com/rhq-project/rhq/commit/4d07cd682e5084f81e4171e1265b9f0c22... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/pom.xml A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/AbortedException.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceIndex.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/TaskTracker.java
Log Message: ----------- [BZ 185375] lots of clean up, refactoring, and perf improvements
The TaskTracker class has been copied into the source tree from the service-metrics maven module. The reason for copying the class is beause the changes for this BZ are going into a patch release, and I want to minimize the number of modules that have to be touched for this.
We no longer query the rdbms to load schedule ids. Instead we use the legacy thrift APIs via the hector library to scan for schedule ids. This way, we query for only those schedule ids that have data.
Reads are now throttled using a RateLimiter. This is better than using a semaphore for a couple reasons. First, we do not have to worry about releasing permits, which had to be done in a couple different places in the code. Secondly, it is more expressive about the throughput. With a semaphore, it was hard to determine the read throughput, whereas with RateLimiter it is directly specified as querys per second.
The migration code previously tried to abort processing if a write failed. This made the code overly complicated. Since we are now using batches, we only do a handful of writes per schedule id. We now just have the failure detection and handling code in the migration finished callback. This simplifies things a lot.
Commit: d4d8743e6364ef9078f40700391d9bcacd2d97ce https://github.com/rhq-project/rhq/commit/d4d8743e6364ef9078f40700391d9bcacd... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message: ----------- [BZ 185375] add failure detection
There is a now a failure threshold that if exceeded will cause the migration to be aborted. The rationale is that the read/write rates will likely have to be adjusted based on each environment. If we are reading too fast for example, and start generating lots of failiures, it makes sense to go ahead and terminate the migration and restart with a lower read rate.
Commit: 966e923f6fff5c939d0df9c1b74bb9019615155e https://github.com/rhq-project/rhq/commit/966e923f6fff5c939d0df9c1b74bb90196... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/pom.xml M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java
Log Message: ----------- [BZ 185375] remove key scanning code with hector
I went back and did some testing with the original approach of querying against all schedule ids that are returned from the RDBMS. I was able to achieve the same performance (if not better) as I did with hector. I do not see any reason therefore to pull in additional dependencies.
Conflicts: modules/common/cassandra-schema/pom.xml
Commit: 68313f3b163e8e627079730ec42bb7e8020b6980 https://github.com/rhq-project/rhq/commit/68313f3b163e8e627079730ec42bb7e802... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/pom.xml M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message: ----------- [BZ 185375] load schedule ids using astyanax
We do not have a good way in C* 1.2 to scan for schedule ids. Instead I am using netflix's astyanax library which uses the legacy thrift api. It is using the get_range_slices operation. This should hopefully be faster than querying every single schedule id returned from the rdbms.
Conflicts: modules/common/cassandra-schema/pom.xml
Commit: c5ae07fd1e2b1f9ced416de148f9ad7d1d77fb6a https://github.com/rhq-project/rhq/commit/c5ae07fd1e2b1f9ced416de148f9ad7d1d... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
Log Message: ----------- [BZ 185375] initial support for dynamic throttling
Commit: f6b2fc9dd3d8cec578c130ebe569cf537337add5 https://github.com/rhq-project/rhq/commit/f6b2fc9dd3d8cec578c130ebe569cf5373... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
Log Message: ----------- [BZ 185375] retry data migrations on failures
Commit: 4d87b270289ce8c402626fc7281c2415f71ee11b https://github.com/rhq-project/rhq/commit/4d87b270289ce8c402626fc7281c2415f7... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceRHQ411Index.java
Log Message: ----------- [BZ 185375] use async writes and dynamic throttling for index update
Commit: cb4b4f38029ae8b83147beebfe30ca1bbadaf266 https://github.com/rhq-project/rhq/commit/cb4b4f38029ae8b83147beebfe30ca1bba... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java M modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/MigrateAggregateMetricsTest.java M modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/SchemaUpgradeTest.java
Log Message: ----------- [BZ 185375] fix regression in MigrateData
Add back check to execute statements when there is not a full batch. Also updating RateMonitor to include minimum rates.
Commit: 90d9837d5bda4e0a3891f15176f49f62320a3f2d https://github.com/rhq-project/rhq/commit/90d9837d5bda4e0a3891f15176f49f6232... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/pom.xml M modules/enterprise/server/appserver/src/main/scripts/rhq-container.build.xml M modules/enterprise/server/installer/src/main/resources/module/main/module.xml M pom.xml
Log Message: ----------- [BZ 185375] adding jboss modules for astyanax
Conflicts: modules/common/cassandra-schema/pom.xml
Commit: e4989b1501113fd4eff56eed2246b95615a089c0 https://github.com/rhq-project/rhq/commit/e4989b1501113fd4eff56eed2246b95615... Author: John Sanda jsanda@redhat.com Date: 2015-03-01 (Sun, 01 Mar 2015)
Changed paths: M modules/common/cassandra-schema/pom.xml M modules/enterprise/server/appserver/src/main/scripts/rhq-container.build.xml
Log Message: ----------- [BZ 185375] fix astyanax module dependencies
Compare: https://github.com/rhq-project/rhq/compare/289920aeb8d8...e4989b150111