Re: Discourse - DeviceMapper causing corruption?

[Philip Rhoades]

People,


>  Date: Mon, 21 Mar 2016 15:02:02 -0600
>  From: Chris Murphy <lists(a)colorremedies.com&gt;
>  To: phil(a)pricom.com.au, 	Community support for Fedora users
>  	<users(a)lists.fedoraproject.org&gt;
>  Subject: Re: Discourse - DeviceMapper causing corruption?
>  Message-ID:
>  	<CAJCQCtSCPQRQsecq0+dzEKdNjEDfBzaDavOOV_e_ARL6LjgkDg(a)mail.gmail.com&gt;
>  Content-Type: text/plain; charset=UTF-8
>  
>  On Mon, Mar 21, 2016 at 1:03 PM, Philip Rhoades <phil(a)pricom.com.au&gt; 
>  wrote:
> > People,
> > 
> > I had a couple of issues to sort out with installing the Docker 
> > Discourse
> > app and while that was being done people made these comments:
> > 
> > "Devicemapper is non starter, fails spectacularly under load and 
> > causes
> > corruption. We block setup if we detect devicemapper. You need aufs or
> > another better supported docker filesystem."
> > 
> > - which was not true - it did install without resorting to aufs.
> > 
> > also:
> > 
> > "Redhat team get very upset when we mention that it just does not work 
> > for
> > us, but release after release they say there are no bugs left, and 
> > each time
> > we keep seeing Discourse users complain about corruption due to device
> > mapper."
> > 
> > Any comments?
>  
>  On the one hand, I'd say, talk is cheap, show me the bug.
>  
>  On the other hand, having blown up thinly provisioned pools quite a
>  few times early on, I basically couldn't handle how non-deterministic
>  the blow ups were, and how difficult the LVM and devicemapper tools
>  are to understand the state of the pool, and fix it. So I gave up and
>  went down the Btrfs path just because I understand it better, and
>  there are only so many rabbit holes one has time and interest to go
>  down.
>  
>  For a good bug report you either need a way to debug it, i.e. you need
>  familiarity with that thing you are debugging. Or you need a concise
>  set of reproduce steps (and a list of versions) so someone else, like
>  a dev or more experienced user, can reproduce that environment and
>  steps and debug it. Without one of those two things at least, I think
>  it's unreasonable to say "hi this is busted" and then just walk away.
>  
>  aufs is just not workable in my opinion because it's not in the
>  mainline kernel. So any sense of portability can't be that important
>  if you're going to depend on aufs. There are plenty of bugs to go
>  around at this point, between overlay(fs), btrfs, and devicemapper
>  although I think by now most of the devicemapper stuff is probably the
>  more stable of the three (?) just a guess really. I had 500+ container
>  states as Btrfs subvolumes and never ran into a Btrfs related problem
>  though. What are the CoreOS folks using? Overlay or dm? 

I had another response on the Discourse forum:

"It's probably this bug, devicemapper seems to be unable to free deleted 
files until you delete the container:

   https://github.com/docker/docker/issues/188672

Which is catastrophic for any long-lived server that uses temporary 
files in any capacity at all."

Does that also make sense to more clued-up tech people than me?

Regards,

Phil.
-- 
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  phil(a)pricom.com.au

Discourse - DeviceMapper causing corruption?

[Chris Murphy]

On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades <phil(a)pricom.com.au&gt; wrote:

>  I had another response on the Discourse forum:
>
>  "It's probably this bug, devicemapper seems to be unable to free deleted
>  files until you delete the container:
>
>     https://github.com/docker/docker/issues/188672
>
>  Which is catastrophic for any long-lived server that uses temporary
>  files in any capacity at all."
>
>  Does that also make sense to more clued-up tech people than me?
>
>
>  That link doesn't work. 
There was a thinp trim bug, but is been squashed on Fedora for over a year.


Chris Murphy

Discourse - DeviceMapper causing corruption?

[Philip Rhoades]

Chris,


On 2016-04-04 03:25, Chris Murphy wrote:
>  On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades
> <phil(a)pricom.com.au&gt;
>  wrote:
>  
> > I had another response on the Discourse forum:
> > 
> > "It's probably this bug, devicemapper seems to be unable to free
> > deleted
> > files until you delete the container:
> > 
> > https://github.com/docker/docker/issues/188672
> > 
> > Which is catastrophic for any long-lived server that uses temporary
> > files in any capacity at all."
> > 
> > Does that also make sense to more clued-up tech people than me?
>  
> > 
>  
>  That link doesn't work. 

Sorry, it looks like copying and pasting added a "2" to the link (the 
number of clicks on the link on the forum) - it should be:

   https://github.com/docker/docker/issues/18867


>  There was a thinp trim bug, but is been squashed on Fedora for over
> a
>  year. 

No mention of "thinp" on the page . .

Thanks,

Phil.
-- 
Philip Rhoades

PO Box 896
Cowra  NSW  2794
Australia
E-mail:  phil(a)pricom.com.au

Discourse - DeviceMapper causing corruption?

[Chris Murphy]

On Sun, Apr 3, 2016 at 5:09 PM, Philip Rhoades <phil(a)pricom.com.au&gt; wrote:
>  Chris,
>
>
>  On 2016-04-04 03:25, Chris Murphy wrote:
> >
> > On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades <phil(a)pricom.com.au&gt;
> > wrote:
> >
> >> I had another response on the Discourse forum:
> >>
> >> "It's probably this bug, devicemapper seems to be unable to free
> >> deleted
> >> files until you delete the container:
> >>
> >> https://github.com/docker/docker/issues/188672
> >>
> >> Which is catastrophic for any long-lived server that uses temporary
> >> files in any capacity at all."
> >>
> >> Does that also make sense to more clued-up tech people than me?
> >
> >
> >>
> >
> > That link doesn't work.
>
>
>
>  Sorry, it looks like copying and pasting added a "2" to the link (the number
>  of clicks on the link on the forum) - it should be:
>
>    https://github.com/docker/docker/issues/18867
>
>
> > There was a thinp trim bug, but is been squashed on Fedora for over a
> > year.
>
>
>
>  No mention of "thinp" on the page . . 

It's using loop devices which is only the default in Docker because
it's easy for them to setup out of the box, where the recommended (at
least on RHEL, CentOS, Fedora) XFS on dmthin requires a.) an LVM VG
with free extents available, and b.) systemctl enable/start
docker-storage-setup to allocate bit more or less than half of those
free extents to the creation of a dmthin pool in which the containers
exist.

What I can't tell you off hand is whether there's something in Docker
land that automatically does discard or fstrim periodically (like on
container deletion) to actually revert the logical extents that are
freed up, back into the dmthin pool.
https://github.com/docker/docker/issues/12902

Looks like something is happening, I'm just not sure what.



-- 
Chris Murphy

Discourse - DeviceMapper causing corruption?

[Chris Murphy]

On Sun, Apr 3, 2016 at 8:20 PM, Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Sun, Apr 3, 2016 at 5:09 PM, Philip Rhoades
> <phil(a)pricom.com.au&gt; wrote:
> > Chris,
> >
> >
> > On 2016-04-04 03:25, Chris Murphy wrote:
> >>
> >> On Fri, Apr 1, 2016, 10:26 PM Philip Rhoades <phil(a)pricom.com.au&gt;
> >> wrote:
> >>
> >>> I had another response on the Discourse forum:
> >>>
> >>> "It's probably this bug, devicemapper seems to be unable to free
> >>> deleted
> >>> files until you delete the container:
> >>>
> >>> https://github.com/docker/docker/issues/188672
> >>>
> >>> Which is catastrophic for any long-lived server that uses temporary
> >>> files in any capacity at all."
> >>>
> >>> Does that also make sense to more clued-up tech people than me?
> >>
> >>
> >>>
> >>
> >> That link doesn't work.
> >
> >
> >
> > Sorry, it looks like copying and pasting added a "2" to the link (the
> number
> > of clicks on the link on the forum) - it should be:
> >
> >   https://github.com/docker/docker/issues/18867
> >
> >
> >> There was a thinp trim bug, but is been squashed on Fedora for over a
> >> year.
> >
> >
> >
> > No mention of "thinp" on the page . .
>
>
>  It's using loop devices which is only the default in Docker because
>  it's easy for them to setup out of the box, where the recommended 
The other thing is Docker on loop device is slow.
http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docke...


cat /usr/bin/docker-storage-setup

The gist is that you want a Volume Group with a decent chunk of free
extents. If you do that, then 'systemctl enable docker-storage-setup'
and then 'systemctl start docker-storage-setup' should do the trick.
You can check it with 'systemctl status docker-storage-setup' to see
if it worked or if there are errors listed.

If you can make semi-decent sense out of that bash script, and you
aren't already familiar with Btrfs, then you should use dmthin and the
docker-storage-setup script. The biggest gotcha is that lvs/lvdisplay
will show the resulting thin pool, but you can't use LVM tools to
manage it. It's strictly a device mapper thin pool. Fortunately, you
shouldn't have to directly interact with it though.

The main reason why I prefer Btrfs is simply because I know where the
bodies are buried, and I don't know where the landmines are in the
device mapper black box. But I'd only recommend Btrfs instead if you
are already somewhat familiar with it, and are comfortable with either
baby sitting the occasional need to balance, or adapting the openSUSE
scripts that can be set on a systemd timer for this purpose.


-- 
Chris Murphy