These are patches I've come up with as part of my school project focused on random data entropy in the installation process. The critical part that needs high-quality random data is disk encryption. Patches 1/3 and 2/3 are, I believe, useful in either case, patch 3/3 adds the entropy check and wait before the LUKS format is created. Related Anaconda patches add a GUI dialog to inform user what is going on and how they could help.
The right value of the minimal entropy level is something experts in that area should tell us. I'll work on that and change it before pushing if these patches get approved.
Vratislav Podzimek (3): Add a way to pass callbacks to storage processing Add callbacks for format creation and resizing Wait for enough entropy when creating LUKS format
blivet/__init__.py | 35 +++++++++++++--- blivet/deviceaction.py | 112 +++++++++++++++++++++++++++++-------------------- blivet/devicetree.py | 6 +-- blivet/formats/luks.py | 1 + blivet/util.py | 4 ++ 5 files changed, 105 insertions(+), 53 deletions(-)
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- blivet/__init__.py | 25 ++++++++++++++++++++----- blivet/deviceaction.py | 20 ++++++++++++-------- blivet/devicetree.py | 6 +++--- 3 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/blivet/__init__.py b/blivet/__init__.py index 2d69eb1..109f7b8 100644 --- a/blivet/__init__.py +++ b/blivet/__init__.py @@ -149,8 +149,14 @@ def storageInitialize(storage, ksdata, protected): if d.name not in ksdata.ignoredisk.ignoredisk] log.debug("onlyuse is now: %s" % (",".join(ksdata.ignoredisk.onlyuse)))
-def turnOnFilesystems(storage, mountOnly=False): - """ Perform installer-specific activation of storage configuration. """ +def turnOnFilesystems(storage, mountOnly=False, callbacks=None): + """ + Perform installer-specific activation of storage configuration. + + :param callbacks: callbacks that should be run (see doIt for more info) + + """ + if not flags.installer_mode: return
@@ -162,7 +168,7 @@ def turnOnFilesystems(storage, mountOnly=False): storage.devicetree.teardownAll()
try: - storage.doIt() + storage.doIt(callbacks=callbacks) except FSResizeError as e: if os.path.exists("/tmp/resize.out"): details = open("/tmp/resize.out", "r").read() @@ -302,8 +308,17 @@ class Blivet(object): self.roots = [] self.services = set()
- def doIt(self): - self.devicetree.processActions() + def doIt(self, callbacks=None): + """ + Do overall storage setup except for writing configuration files. + + :param callbacks: a dictionary mapping action keywords to functions, + that should be called (see below for more info) + :type callbacks: dict (str -> function) + + """ + + self.devicetree.processActions(callbacks=callbacks) self.doEncryptionPassphraseRetrofits()
# now set the boot partition's flag diff --git a/blivet/deviceaction.py b/blivet/deviceaction.py index 202a66c..d139436 100644 --- a/blivet/deviceaction.py +++ b/blivet/deviceaction.py @@ -160,8 +160,12 @@ class DeviceAction(object): self.id = DeviceAction._id DeviceAction._id += 1
- def execute(self): - """ perform the action """ + def execute(self, callbacks=None): + """ + perform the action + :param callbacks: see Blivet.doIt + + """ pass
def cancel(self): @@ -268,7 +272,7 @@ class ActionCreateDevice(DeviceAction): # FIXME: assert device.fs is None DeviceAction.__init__(self, device)
- def execute(self): + def execute(self, callbacks=None): self.device.create()
def requires(self, action): @@ -314,7 +318,7 @@ class ActionDestroyDevice(DeviceAction): if device.exists: device.teardown()
- def execute(self): + def execute(self, callbacks=None): self.device.destroy()
# Make sure libparted does not keep cached info for this device @@ -399,7 +403,7 @@ class ActionResizeDevice(DeviceAction):
self.device.targetSize = newsize
- def execute(self): + def execute(self, callbacks=None): self.device.resize()
def cancel(self): @@ -447,7 +451,7 @@ class ActionCreateFormat(DeviceAction): else: self.origFormat = getFormat(None)
- def execute(self): + def execute(self, callbacks=None): msg = _("Creating %(type)s on %(device)s") % {"type": self.device.format.type, "device": self.device.path} with progress_report(msg): self.device.setup() @@ -521,7 +525,7 @@ class ActionDestroyFormat(DeviceAction): device.format.teardown() self.device.format = None
- def execute(self): + def execute(self, callbacks=None): """ wipe the filesystem signature from the device """ self.device.setup(orig=True) self.format.destroy() @@ -588,7 +592,7 @@ class ActionResizeFormat(DeviceAction): self.origSize = self.device.format.targetSize self.device.format.targetSize = newsize
- def execute(self): + def execute(self, callbacks=None): msg = _("Resizing filesystem on %(device)s") % {"device": self.device.path} with progress_report(msg): self.device.setup(orig=True) diff --git a/blivet/devicetree.py b/blivet/devicetree.py index 9159d9e..8707616 100644 --- a/blivet/devicetree.py +++ b/blivet/devicetree.py @@ -185,7 +185,7 @@ class DeviceTree(object): actions.append(self._actions[idx]) self._actions = actions
- def processActions(self, dryRun=None): + def processActions(self, dryRun=None, callbacks=None): """ Execute all registered actions. """ log.info("resetting parted disks...") for device in self.devices: @@ -234,12 +234,12 @@ class DeviceTree(object): log.info("executing action: %s" % action) if not dryRun: try: - action.execute() + action.execute(callbacks) except DiskLabelCommitError: # it's likely that a previous format destroy action # triggered setup of an lvm or md device. self.teardownAll() - action.execute() + action.execute(callbacks)
udev_settle() for device in self._devices:
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- blivet/__init__.py | 9 ++++++ blivet/deviceaction.py | 85 +++++++++++++++++++++++++------------------------- 2 files changed, 52 insertions(+), 42 deletions(-)
diff --git a/blivet/__init__.py b/blivet/__init__.py index 109f7b8..13cb523 100644 --- a/blivet/__init__.py +++ b/blivet/__init__.py @@ -316,6 +316,15 @@ class Blivet(object): that should be called (see below for more info) :type callbacks: dict (str -> function)
+ For now the following callbacks are supported (keywords and arguments + the called function gets): + + { "CreateFormatPre": (msg), + "CreateFormatPost": (msg), + "ResizeFormatPre": (msg), + "ResizeFormatPost": (msg), + } + """
self.devicetree.processActions(callbacks=callbacks) diff --git a/blivet/deviceaction.py b/blivet/deviceaction.py index d139436..7a162cd 100644 --- a/blivet/deviceaction.py +++ b/blivet/deviceaction.py @@ -37,17 +37,6 @@ _ = lambda x: gettext.ldgettext("blivet", x) import logging log = logging.getLogger("blivet")
-from contextlib import contextmanager - -@contextmanager -def progress_report_stub(message): - yield - -try: - from pyanaconda.progress import progress_report -except ImportError: - progress_report = progress_report_stub - # The values are just hints as to the ordering. # Eg: fsmod and devmod ordering depends on the mod (shrink -v- grow) ACTION_TYPE_NONE = 0 @@ -452,33 +441,39 @@ class ActionCreateFormat(DeviceAction): self.origFormat = getFormat(None)
def execute(self, callbacks=None): - msg = _("Creating %(type)s on %(device)s") % {"type": self.device.format.type, "device": self.device.path} - with progress_report(msg): - self.device.setup() - - if isinstance(self.device, PartitionDevice): - for flag in partitionFlag.keys(): - # Keep the LBA flag on pre-existing partitions - if flag in [ PARTITION_LBA, self.format.partedFlag ]: - continue - self.device.unsetFlag(flag) - - if self.format.partedFlag is not None: - self.device.setFlag(self.format.partedFlag) - - if self.format.partedSystem is not None: - self.device.partedPartition.system = self.format.partedSystem - - self.device.disk.format.commitToDisk() - - self.device.format.create(device=self.device.path, - options=self.device.formatArgs) - # Get the UUID now that the format is created - udev_settle() - self.device.updateSysfsPath() - info = udev_get_block_device(self.device.sysfsPath) - self.device.format.uuid = udev_device_get_uuid(info) - self.device.deviceLinks = udev_device_get_symlinks(info) + if callbacks and "CreateFormatPre" in callbacks: + msg = _("Creating %(type)s on %(device)s") % {"type": self.device.format.type, "device": self.device.path} + callbacks["CreateFormatPre"](msg) + + self.device.setup() + + if isinstance(self.device, PartitionDevice): + for flag in partitionFlag.keys(): + # Keep the LBA flag on pre-existing partitions + if flag in [ PARTITION_LBA, self.format.partedFlag ]: + continue + self.device.unsetFlag(flag) + + if self.format.partedFlag is not None: + self.device.setFlag(self.format.partedFlag) + + if self.format.partedSystem is not None: + self.device.partedPartition.system = self.format.partedSystem + + self.device.disk.format.commitToDisk() + + self.device.format.create(device=self.device.path, + options=self.device.formatArgs) + # Get the UUID now that the format is created + udev_settle() + self.device.updateSysfsPath() + info = udev_get_block_device(self.device.sysfsPath) + self.device.format.uuid = udev_device_get_uuid(info) + self.device.deviceLinks = udev_device_get_symlinks(info) + + if callbacks and "CreateFormatPost" in callbacks: + msg = _("Created %(type)s on %(device)s") % {"type": self.device.format.type, "device": self.device.path} + callbacks["CreateFormatPost"](msg)
def cancel(self): self.device.format = self.origFormat @@ -593,10 +588,16 @@ class ActionResizeFormat(DeviceAction): self.device.format.targetSize = newsize
def execute(self, callbacks=None): - msg = _("Resizing filesystem on %(device)s") % {"device": self.device.path} - with progress_report(msg): - self.device.setup(orig=True) - self.device.format.doResize() + if callbacks and "ResizeFormatPre" in callbacks: + msg = _("Resizing filesystem on %(device)s") % {"device": self.device.path} + callbacks["ResizeFormatPre"](msg) + + self.device.setup(orig=True) + self.device.format.doResize() + + if callbacks and "ResizeFormatPost" in callbacks: + msg = _("Resized filesystem on %(device)s") % {"device": self.device.path} + callbacks["ResizeFormatPost"](msg)
def cancel(self): self.device.format.targetSize = self.origSize
Also invoke the callback for such event (if any).
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- blivet/__init__.py | 3 ++- blivet/deviceaction.py | 17 +++++++++++++++++ blivet/formats/luks.py | 1 + blivet/util.py | 4 ++++ 4 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/blivet/__init__.py b/blivet/__init__.py index 13cb523..0751be8 100644 --- a/blivet/__init__.py +++ b/blivet/__init__.py @@ -316,13 +316,14 @@ class Blivet(object): that should be called (see below for more info) :type callbacks: dict (str -> function)
- For now the following callbacks are supported (keywords and arguments + For now, the following callbacks are supported (keywords and arguments the called function gets):
{ "CreateFormatPre": (msg), "CreateFormatPost": (msg), "ResizeFormatPre": (msg), "ResizeFormatPost": (msg), + "WaitForEntropy": (msg, min_entropy), }
""" diff --git a/blivet/deviceaction.py b/blivet/deviceaction.py index 7a162cd..b0b42e0 100644 --- a/blivet/deviceaction.py +++ b/blivet/deviceaction.py @@ -23,12 +23,16 @@
from udev import * import math +import time
from devices import StorageDevice from devices import PartitionDevice from devices import LVMLogicalVolumeDevice from formats import getFormat from errors import * +from util import get_current_entropy +from formats import luks + from parted import partitionFlag, PARTITION_LBA
import gettext @@ -462,6 +466,19 @@ class ActionCreateFormat(DeviceAction):
self.device.disk.format.commitToDisk()
+ if isinstance(self.device.format, luks.LUKS): + # LUKS needs to wait for random data entropy if it is too low + current_entropy = get_current_entropy() + if current_entropy < luks.MINIMAL_ENTROPY: + if callbacks and "WaitForEntropy" in callbacks: + msg = "Not enough entropy to create LUKS format. "\ + "%d bits are needed." % luks.MINIMAL_ENTROPY + callbacks["WaitForEntropy"](msg, luks.MINIMAL_ENTROPY) + + while get_current_entropy() < luks.MINIMAL_ENTROPY: + # wait for entropy to become high enough + time.sleep(1) + self.device.format.create(device=self.device.path, options=self.device.formatArgs) # Get the UUID now that the format is created diff --git a/blivet/formats/luks.py b/blivet/formats/luks.py index ab12eba..1a9d836 100644 --- a/blivet/formats/luks.py +++ b/blivet/formats/luks.py @@ -40,6 +40,7 @@ _ = lambda x: gettext.ldgettext("blivet", x) import logging log = logging.getLogger("blivet")
+MINIMAL_ENTROPY = 256
class LUKS(DeviceFormat): """ A LUKS device. """ diff --git a/blivet/util.py b/blivet/util.py index 3e51249..ca4ae0c 100644 --- a/blivet/util.py +++ b/blivet/util.py @@ -326,3 +326,7 @@ def insert_colons(a_string): return insert_colons(a_string[:-2]) + ':' + suffix else: return suffix + +def get_current_entropy(): + with open("/proc/sys/kernel/random/entropy_avail", "r") as fobj: + return int(fobj.readline())
On Fri, 2013-05-24 at 19:01 +0200, Vratislav Podzimek wrote:
Also invoke the callback for such event (if any).
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
Good idea. The blivet set looks good in general, but I have a few comments:
1. MINIMAL_ENTROPY should go into devicelibs.crypto instead of formats.luks 2. consider moving get_current_entropy into devicelibs.crypto 3. it would be nice if there were some abstraction of the callback handling in the DeviceAction classes
On Fri, 2013-05-24 at 12:20 -0500, David Lehman wrote:
On Fri, 2013-05-24 at 19:01 +0200, Vratislav Podzimek wrote:
Also invoke the callback for such event (if any).
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
Good idea. The blivet set looks good in general, but I have a few comments:
- MINIMAL_ENTROPY should go into devicelibs.crypto instead of formats.luks
- consider moving get_current_entropy into devicelibs.crypto
Will do, thanks for the tips. I was thinking about where to puth these things, but I've overseen the devicelibs.crypto module.
- it would be nice if there were some abstraction of the callback handling in the DeviceAction classes
Yeah, I agree with that. But I'm not sure there is any nice way how to do it. I was thinking about all actions calling their "pre" and "post" callbacks with names derived from their class names, but that would be a hack, I think. Any better suggestions? Maybe adding some _pre_callback and _post_callback class attributes holding the names of the callbacks to the actions?
- it would be nice if there were some abstraction of the callback handling in the DeviceAction classes
Yeah, I agree with that. But I'm not sure there is any nice way how to do it. I was thinking about all actions calling their "pre" and "post" callbacks with names derived from their class names, but that would be a hack, I think. Any better suggestions? Maybe adding some _pre_callback and _post_callback class attributes holding the names of the callbacks to the actions?
Do you envision adding a bunch more callbacks later? Do you envison that other consumers of blivet will want different sets of callbacks? You could perhaps do a base class that defines callback methods, then a subclass for your specific callbacks, then instantiate that.
- Chris
On Tue, 2013-05-28 at 11:02 -0400, Chris Lumens wrote:
- it would be nice if there were some abstraction of the callback handling in the DeviceAction classes
Yeah, I agree with that. But I'm not sure there is any nice way how to do it. I was thinking about all actions calling their "pre" and "post" callbacks with names derived from their class names, but that would be a hack, I think. Any better suggestions? Maybe adding some _pre_callback and _post_callback class attributes holding the names of the callbacks to the actions?
Do you envision adding a bunch more callbacks later? Do you envison that other consumers of blivet will want different sets of callbacks? You could perhaps do a base class that defines callback methods, then a subclass for your specific callbacks, then instantiate that.
That might be the right way to go, thanks! I'll try to think it through and suggest a modified version of these patches.
On Fri, 2013-05-24 at 19:01 +0200, Vratislav Podzimek wrote:
diff --git a/blivet/formats/luks.py b/blivet/formats/luks.py index ab12eba..1a9d836 100644 --- a/blivet/formats/luks.py +++ b/blivet/formats/luks.py @@ -40,6 +40,7 @@ _ = lambda x: gettext.ldgettext("blivet", x) import logging log = logging.getLogger("blivet")
+MINIMAL_ENTROPY = 256
The default luks key size we use is 512 bits, although I'm not sure how/if that factors into this default value of MINIMAL_ENTROPY.
@@ -462,6 +466,19 @@ class ActionCreateFormat(DeviceAction):
self.device.disk.format.commitToDisk()
if isinstance(self.device.format, luks.LUKS):# LUKS needs to wait for random data entropy if it is too lowcurrent_entropy = get_current_entropy()if current_entropy < luks.MINIMAL_ENTROPY:if callbacks and "WaitForEntropy" in callbacks:msg = "Not enough entropy to create LUKS format. "\"%d bits are needed." % luks.MINIMAL_ENTROPYcallbacks["WaitForEntropy"](msg, luks.MINIMAL_ENTROPY)while get_current_entropy() < luks.MINIMAL_ENTROPY:# wait for entropy to become high enoughtime.sleep(1)self.device.format.create(device=self.device.path, options=self.device.formatArgs) # Get the UUID now that the format is created
Would it also be possible to somehow unpredictably stir things up here to increase available entropy? That could help the loop take less time to complete.
- Chris
On Tue, 2013-05-28 at 11:00 -0400, Chris Lumens wrote:
@@ -462,6 +466,19 @@ class ActionCreateFormat(DeviceAction):
self.device.disk.format.commitToDisk()
if isinstance(self.device.format, luks.LUKS):# LUKS needs to wait for random data entropy if it is too lowcurrent_entropy = get_current_entropy()if current_entropy < luks.MINIMAL_ENTROPY:if callbacks and "WaitForEntropy" in callbacks:msg = "Not enough entropy to create LUKS format. "\"%d bits are needed." % luks.MINIMAL_ENTROPYcallbacks["WaitForEntropy"](msg, luks.MINIMAL_ENTROPY)while get_current_entropy() < luks.MINIMAL_ENTROPY:# wait for entropy to become high enoughtime.sleep(1)self.device.format.create(device=self.device.path, options=self.device.formatArgs) # Get the UUID now that the format is createdWould it also be possible to somehow unpredictably stir things up here to increase available entropy? That could help the loop take less time to complete.
I was thinking about leaving that to the callback (as the Anaconda's one do), but maybe there could be done something even in the loop. However, it's quite hard to come up with something "unpredictable". If we had anything like that we could directly populate the kernel's pool. The only thing that comes to my mind is some "quite random" reading from disks, that would generate entropy through seek times and so on, but I'm not sure anything like would be possible.
But the most important thing is that the entropy of e.g. 256 bits (I'm still waiting for the right value of the threshold) should be already available or gathered quite quickly.
On Fri, May 24, 2013 at 07:01:41PM +0200, Vratislav Podzimek wrote:
These are patches I've come up with as part of my school project focused on random data entropy in the installation process. The critical part that needs high-quality random data is disk encryption. Patches 1/3 and 2/3 are, I believe, useful in either case, patch 3/3 adds the entropy check and wait before the LUKS format is created. Related Anaconda patches add a GUI dialog to inform user what is going on and how they could help.
One thing I worry about is a kickstart being starved for entropy. Do you want to hang for however long it takes, do you want to disable waiting or provide a maximum timeout?
Depending on the hardware being used you will have more (or less) entropy available. Last time I looked some of the network drivers contributed, but not all of them.
On Tue, 2013-05-28 at 10:26 -0700, Brian C. Lane wrote:
On Fri, May 24, 2013 at 07:01:41PM +0200, Vratislav Podzimek wrote:
These are patches I've come up with as part of my school project focused on random data entropy in the installation process. The critical part that needs high-quality random data is disk encryption. Patches 1/3 and 2/3 are, I believe, useful in either case, patch 3/3 adds the entropy check and wait before the LUKS format is created. Related Anaconda patches add a GUI dialog to inform user what is going on and how they could help.
One thing I worry about is a kickstart being starved for entropy. Do you want to hang for however long it takes, do you want to disable waiting or provide a maximum timeout?
Good question. There might be some maximum timeout needed. However, I think that the right threshold (which I'm still waiting for) should be low enough not to hit any issues like that.
Depending on the hardware being used you will have more (or less) entropy available. Last time I looked some of the network drivers contributed, but not all of them.
It's network drivers, HDD operations, keyboard, mouse, special instructions HW, special instructions provided by the Ivy Bridge processors etc.
anaconda-patches@lists.fedorahosted.org
-
Brian C. Lane -
Chris Lumens -
David Lehman -
Vratislav Podzimek