- First time I have seen a Dto object in the code base. What caused us to need it? - Who calls the ctor on CertificateRevocationListTask? How does the parameters get set? - Reading in the entire CRL into memory scares me :) If we assume a serial number is a long of average length stored as a string, then it is 10 bytes. Add another 18 bytes for the time stamp as a string, and then you get 28 bytes per entry. A million entitlements would get us roughly a 26 meg file. Is there a way to stream this? Perhaps read in each record from the CRL and the process it into the new file? - Related to this, lets turn down the logging :) - The logic for deleting the old certificates is to delete those that expired yesterday certs. Are there any rules about how long a cert needs to be in a CRL? - Does the CRL need to be signed? - Have we tried loading the CRL into apache or an ocspd daemon to see it will work?
-- bk
On Fri, Jul 2, 2010 at 1:16 PM, Bryan Kearney bkearney@redhat.com wrote:
- First time I have seen a Dto object in the code base. What caused us
to need it?
Not me! :D
- Who calls the ctor on CertificateRevocationListTask? How does the
parameters get set?
HighlanderJobFactory takes care of configuring the tasks as well as injecting dependencies into the task objects.
- Reading in the entire CRL into memory scares me :) If we assume a
serial number is a long of average length stored as a string, then it is 10 bytes. Add another 18 bytes for the time stamp as a string, and then you get 28 bytes per entry. A million entitlements would get us roughly a 26 meg file. Is there a way to stream this? Perhaps read in each record from the CRL and the process it into the new file?
I had the same concern and I concur with you 100%. I have gone through some of the approaches below.
1. Streaming api is not provided by bouncycastle(AFAIK). I think streaming api would have done lot of help to us considering the fact that the number of revoked certificates does not go down. 2. However, we can reduce our memory consumption by creating 'delta CRL' . As the name suggests, delta CRL contains only 'new serial numbers' when compared to the 'base CRL'. Base CRL is usually a big CRL containing as much as possible serial number entries and delta CRL are generated every day or perhaps even sooner. From what I gathered, 'base CRL' is generated at the beginning of every week and delta is generated every day(or sooner). So this cycle is repeated every week. If you look at it, even though delta crl help us reduce our memory consumption, as time passes on, the number of delta crl files tend to increase and we would have to generate the base CRL once in few weeks/some_interval. But when we generate this base CRL, we would have to load all the serial numbers from the database into the memory. Once again we are back to the same memory issue! 3. CRL can also be generated by partitioning them based on some logic say 1000 serials per cert. Later on, when removing old/expired serials, we iterate over each certificate and update. This will reduce our memory consumption and keep the code clean. However, the number of certificates generated would be large and the logic to generate new crl would be confusing as well.(when do we generate a new crl? everyday? or generate a new crl only if the current crl has exceeded 1000 entries...).
Also in general another concern with splitting crl is how its being handled by the servers and whether they support it. I gathered all of these from the internet/security books. I am sure people having real experience with the crl stuff would have more practical knowledge as well as better approaches.
- Related to this, lets turn down the logging :)
I introduced logging for testing purpose. I will turn them down to trace level.
- The logic for deleting the old certificates is to delete those that
expired yesterday certs. Are there any rules about how long a cert needs to be in a CRL?
- Does the CRL need to be signed?
- Have we tried loading the CRL into apache or an ocspd daemon to see it
will work?
-- bk
candlepin mailing list candlepin@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/candlepin
On 07/06/2010 11:13 AM, Ajay Kumar wrote:
On Fri, Jul 2, 2010 at 1:16 PM, Bryan Kearneybkearney@redhat.com wrote:
- First time I have seen a Dto object in the code base. What caused us
to need it?
Not me! :D
- Who calls the ctor on CertificateRevocationListTask? How does the
parameters get set?
HighlanderJobFactory takes care of configuring the tasks as well as injecting dependencies into the task objects.
- Reading in the entire CRL into memory scares me :) If we assume a
serial number is a long of average length stored as a string, then it is 10 bytes. Add another 18 bytes for the time stamp as a string, and then you get 28 bytes per entry. A million entitlements would get us roughly a 26 meg file. Is there a way to stream this? Perhaps read in each record from the CRL and the process it into the new file?
I had the same concern and I concur with you 100%. I have gone through some of the approaches below.
- Streaming api is not provided by bouncycastle(AFAIK). I think
streaming api would have done lot of help to us considering the fact that the number of revoked certificates does not go down.
This is what i was thinking. Instead of reading the whole file into memory, could you read on line at a time and then figure out if that line is still valid before writing it out? It may be less efficient, but it would save on memory.
- However, we can reduce our memory consumption by creating 'delta
CRL' . As the name suggests, delta CRL contains only 'new serial numbers' when compared to the 'base CRL'. Base CRL is usually a big CRL containing as much as possible serial number entries and delta CRL are generated every day or perhaps even sooner. From what I gathered, 'base CRL' is generated at the beginning of every week and delta is generated every day(or sooner). So this cycle is repeated every week.
Is this something which ocspd supports?
If you look at it, even though delta crl help us reduce our
memory consumption, as time passes on, the number of delta crl files tend to increase and we would have to generate the base CRL once in few weeks/some_interval. But when we generate this base CRL, we would have to load all the serial numbers from the database into the memory. Once again we are back to the same memory issue! 3. CRL can also be generated by partitioning them based on some logic say 1000 serials per cert. Later on, when removing old/expired serials, we iterate over each certificate and update. This will reduce our memory consumption and keep the code clean. However, the number of certificates generated would be large and the logic to generate new crl would be confusing as well.(when do we generate a new crl? everyday? or generate a new crl only if the current crl has exceeded 1000 entries...).
Also in general another concern with splitting crl is how its being handled by the servers and whether they support it. I gathered all of these from the internet/security books. I am sure people having real experience with the crl stuff would have more practical knowledge as well as better approaches.
Brenton would have hte most viision into this areas.
- Related to this, lets turn down the logging :)
I introduced logging for testing purpose. I will turn them down to trace level.
- The logic for deleting the old certificates is to delete those that
expired yesterday certs. Are there any rules about how long a cert needs to be in a CRL?
- Does the CRL need to be signed?
- Have we tried loading the CRL into apache or an ocspd daemon to see it
will work?
this last one will be a good test methinks :)
--b k
On Tue, Jul 6, 2010 at 11:13 AM, Ajay Kumar ajaykumarns@gmail.com wrote:
On Fri, Jul 2, 2010 at 1:16 PM, Bryan Kearney bkearney@redhat.com wrote:
- First time I have seen a Dto object in the code base. What caused us
to need it?
Not me! :D
- Who calls the ctor on CertificateRevocationListTask? How does the
parameters get set?
HighlanderJobFactory takes care of configuring the tasks as well as injecting dependencies into the task objects.
- Reading in the entire CRL into memory scares me :) If we assume a
serial number is a long of average length stored as a string, then it is 10 bytes. Add another 18 bytes for the time stamp as a string, and then you get 28 bytes per entry. A million entitlements would get us roughly a 26 meg file. Is there a way to stream this? Perhaps read in each record from the CRL and the process it into the new file?
I had the same concern and I concur with you 100%. I have gone through some of the approaches below.
- Streaming api is not provided by bouncycastle(AFAIK). I think
streaming api would have done lot of help to us considering the fact that the number of revoked certificates does not go down.
Is this something we can write and possibly submit back to bouncycastle?
jesus
I believe they don't provide a streaming api because of the fact that if you are going to sign the certificate, you would have to anyway load the entire content into memory(hash/do some security stuff on contents) and sign it. I could be wrong though.
On Tue, Jul 6, 2010 at 11:24 AM, Jesus M. Rodriguez jmrodri@gmail.com wrote:
On Tue, Jul 6, 2010 at 11:13 AM, Ajay Kumar ajaykumarns@gmail.com wrote:
On Fri, Jul 2, 2010 at 1:16 PM, Bryan Kearney bkearney@redhat.com wrote:
- First time I have seen a Dto object in the code base. What caused us
to need it?
Not me! :D
- Who calls the ctor on CertificateRevocationListTask? How does the
parameters get set?
HighlanderJobFactory takes care of configuring the tasks as well as injecting dependencies into the task objects.
- Reading in the entire CRL into memory scares me :) If we assume a
serial number is a long of average length stored as a string, then it is 10 bytes. Add another 18 bytes for the time stamp as a string, and then you get 28 bytes per entry. A million entitlements would get us roughly a 26 meg file. Is there a way to stream this? Perhaps read in each record from the CRL and the process it into the new file?
I had the same concern and I concur with you 100%. I have gone through some of the approaches below.
- Streaming api is not provided by bouncycastle(AFAIK). I think
streaming api would have done lot of help to us considering the fact that the number of revoked certificates does not go down.
Is this something we can write and possibly submit back to bouncycastle?
jesus _______________________________________________ candlepin mailing list candlepin@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/candlepin
candlepin@lists.fedorahosted.org