I suspect this request might fall short, but I figure I'll start a dialog.
With Fedora CoreOS needing some different services set up and more and more requests getting opened every day on the infra issue tracker, it would be nice if I could gain some more access within Fedora Infra to be able to help support the Fedora CoreOS efforts as well as possibly help lighten the load (i.e. perform simple privileged tasks for unprivileged people when appropriate) of Fedora's infra team.
I've had a few services that I recently got all the way through the RFR process. I was admittedly out for a month during that time on leave, but if you subtract that month it still took a rather long time to make it to completion. The long times aren't necessarily anyone's fault, just happens to be how the requests landed and also my projects not necessarily being on anyone's priority list.
That being said, I have learned a lot by getting these apps into prod and I feel like I can help our future apps along if I had more access to debug things myself (i.e. inconsistencies in stage) rather than waiting on someone from infra.
I am not requesting this access to abuse it. I am typically conservative and don't do things that I'm not comfortable doing or that I know would be frowned upon. As an example I was granted access to the releng group a while back and I think that has worked out well.
I'd like to help if possible, but I understand if my request isn't granted.
Dusty
On Fri, 16 Aug 2019 at 16:40, Dusty Mabe dusty@dustymabe.com wrote:
I suspect this request might fall short, but I figure I'll start a dialog.
...
I'd like to help if possible, but I understand if my request isn't granted.
I have a counter idea which probably won't fly well either. My main thing I would like is that people who are in main are doing the following: 1. Regularly taking IRC oncall for a week 2. Regularly taking 'pager' duty for a week. At the moment this means taking all the pages all the time but just listening to it when oncall. I would love a way that pages was just 1-2 people for a week.. which would make that more possible. 3. Let people know what their areas of expertise are and take those tickets.
I am less inclined to have people on main who are just doing it for one set of things because it doesn't help us lower our overall workload.. it just allows that person fix their problem.
On 8/16/19 5:04 PM, Stephen John Smoogen wrote:
On Fri, 16 Aug 2019 at 16:40, Dusty Mabe dusty@dustymabe.com wrote:
I suspect this request might fall short, but I figure I'll start a dialog.
...
I'd like to help if possible, but I understand if my request isn't granted.
I have a counter idea which probably won't fly well either. My main thing I would like is that people who are in main are doing the following:
- Regularly taking IRC oncall for a week
- Regularly taking 'pager' duty for a week.At the moment this means
taking all the pages all the time but just listening to it when oncall. I would love a way that pages was just 1-2 people for a week.. which would make that more possible. 3. Let people know what their areas of expertise are and take those tickets.
I am less inclined to have people on main who are just doing it for one set of things because it doesn't help us lower our overall workload.. it just allows that person fix their problem.
Honestly this may help address my problem as well, just approaching it from a different direction. Having more people around who are capable of "doing things" would certainly help. I guess my proposal includes me being around and capable of doing things, but "yes" with a focus on FCOS related tasks.
On 8/16/19 2:12 PM, Dusty Mabe wrote:
On 8/16/19 5:04 PM, Stephen John Smoogen wrote:
On Fri, 16 Aug 2019 at 16:40, Dusty Mabe dusty@dustymabe.com wrote:
I suspect this request might fall short, but I figure I'll start a dialog.
...
I'd like to help if possible, but I understand if my request isn't granted.
I have a counter idea which probably won't fly well either. My main thing I would like is that people who are in main are doing the following:
- Regularly taking IRC oncall for a week
- Regularly taking 'pager' duty for a week.At the moment this means
taking all the pages all the time but just listening to it when oncall. I would love a way that pages was just 1-2 people for a week.. which would make that more possible. 3. Let people know what their areas of expertise are and take those tickets.
I am less inclined to have people on main who are just doing it for one set of things because it doesn't help us lower our overall workload.. it just allows that person fix their problem.
Honestly this may help address my problem as well, just approaching it from a different direction. Having more people around who are capable of "doing things" would certainly help. I guess my proposal includes me being around and capable of doing things, but "yes" with a focus on FCOS related tasks.
Well, this I think is related to our "agile transformation" (or if you prefer "reorganizing how we work"). Right now, you find it hard to get stuff done because everyone is busy on other things, so you have to nag us to get stuff done which makes things worse for us, etc.
Ideally I think we can get to a place where we do a lot more scheduling and a lot less having to yell for cycles. You note a few things were just 5 min for someone to do... but the way we are working now someone does that thing for you finally, but then goes back to the other stuff they were working on, meaning in 30min you need another thing... repeat.
If you can come to us and say "hey, this is my project, I need it deployed by X" we can look at it, gather info we need, put it in our queue and then tell you "hey, we can work on that next tuesday". Then next tuesday we can devote a block of time to getting everything done. I think thats a win for everyone in the end and I would really like to get there. Can we? I sure hope so.
In fact perhaps we could start doing this now somewhat: block off say... wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
Anyhow, I think the answer is in working better/smarter rather than granting people more perms in general. Of course increasing the pool of people who can do specific things might also help, but also those people will need to learn when to do things and when to ask about doing things.
kevin
On 8/19/19 6:09 PM, Kevin Fenzi wrote:
Well, this I think is related to our "agile transformation" (or if you prefer "reorganizing how we work"). Right now, you find it hard to get stuff done because everyone is busy on other things, so you have to nag us to get stuff done which makes things worse for us, etc.
Right. It has been this way for a long time. It's not your fault and it's not my fault, but both of us feel like we've got the short end of the stick.
Ideally I think we can get to a place where we do a lot more scheduling and a lot less having to yell for cycles. You note a few things were just 5 min for someone to do... but the way we are working now someone does that thing for you finally, but then goes back to the other stuff they were working on, meaning in 30min you need another thing... repeat.
If you can come to us and say "hey, this is my project, I need it deployed by X" we can look at it, gather info we need, put it in our queue and then tell you "hey, we can work on that next tuesday". Then next tuesday we can devote a block of time to getting everything done. I think thats a win for everyone in the end and I would really like to get there. Can we? I sure hope so
In fact perhaps we could start doing this now somewhat: block off say... wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
I think blocking off time and doing some scheduling is a great idea. My advice here is that we have a shared calendar with blocks that people can sign up for automatically (i.e. if the block is open anyone can reserve it as long as they have a FAS account).
The only counter point that I have is that sometimes a "block of time" isn't always what's needed. Sometimes it will be something very small, but rather fundamental, that requires quite a bit of rework by the end user. For example, you and I have a 5 minutes session where some revelation comes to light and I need to go rewrite a portion of my application. I then spend two hours doing that and have to wait a week for the next block of time.
One thing that we might be able to do for that is have separate blocks of time the next day that is specifically for follow ups to the previous day.
Anyhow, I think the answer is in working better/smarter rather than granting people more perms in general. Of course increasing the pool of people who can do specific things might also help, but also those people will need to learn when to do things and when to ask about doing things.
I think working smarter is definitely going to be what we need to do over time to support the rest of Fedora and "initiatives" better. For Fedora CoreOS, for right now I really do think it would help if one of the Fedora CoreOS team were given more access so we can self service better. Right now we either need more access or we need more dedicated attention for the various needs we have. If we could get a one hour a day scheduled commitment I think that would help us quite a bit. We won't always need help (i.e. we cancel) and we won't always need the whole hour, but right now it would really make a difference in our progress.
WDYT?
Dusty
On Tue, 20 Aug 2019 at 15:30, Dusty Mabe dusty@dustymabe.com wrote:
On 8/19/19 6:09 PM, Kevin Fenzi wrote:
Well, this I think is related to our "agile transformation" (or if you prefer "reorganizing how we work"). Right now, you find it hard to get stuff done because everyone is busy on other things, so you have to nag us to get stuff done which makes things worse for us, etc.
Right. It has been this way for a long time. It's not your fault and it's not my fault, but both of us feel like we've got the short end of the stick.
Ideally I think we can get to a place where we do a lot more scheduling and a lot less having to yell for cycles. You note a few things were just 5 min for someone to do... but the way we are working now someone does that thing for you finally, but then goes back to the other stuff they were working on, meaning in 30min you need another thing... repeat.
If you can come to us and say "hey, this is my project, I need it deployed by X" we can look at it, gather info we need, put it in our queue and then tell you "hey, we can work on that next tuesday". Then next tuesday we can devote a block of time to getting everything done. I think thats a win for everyone in the end and I would really like to get there. Can we? I sure hope so
In fact perhaps we could start doing this now somewhat: block off say... wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
I think blocking off time and doing some scheduling is a great idea. My advice here is that we have a shared calendar with blocks that people can sign up for automatically (i.e. if the block is open anyone can reserve it as long as they have a FAS account).
The only counter point that I have is that sometimes a "block of time" isn't always what's needed. Sometimes it will be something very small, but rather fundamental, that requires quite a bit of rework by the end user. For example, you and I have a 5 minutes session where some revelation comes to light and I need to go rewrite a portion of my application. I then spend two hours doing that and have to wait a week for the next block of time.
I think this is the symptom of a lack of upfront cooperation and design. This is what we are currently putting in place, if you need something consequent from us you come up with a spec that explain the WHAT and the WHY. Then we work together on the HOW and WHEN, once we have a good idea of the HOW and WHEN the work will be prioritized on our side (might have a dedicated team assigned to it) which I think will make cooperation easier.
It is never too late to start, so maybe we should spend some time to write down everything that is needed for Fedora CoreOS, write down HOW this will be done and what the work being Done means. That way we could probably dedicated a few people to focus on that work until completion.
How does that sounds ?
One thing that we might be able to do for that is have separate blocks of time the next day that is specifically for follow ups to the previous day.
Anyhow, I think the answer is in working better/smarter rather than granting people more perms in general. Of course increasing the pool of people who can do specific things might also help, but also those people will need to learn when to do things and when to ask about doing things.
I think working smarter is definitely going to be what we need to do over time to support the rest of Fedora and "initiatives" better. For Fedora CoreOS, for right now I really do think it would help if one of the Fedora CoreOS team were given more access so we can self service better. Right now we either need more access or we need more dedicated attention for the various needs we have. If we could get a one hour a day scheduled commitment I think that would help us quite a bit. We won't always need help (i.e. we cancel) and we won't always need the whole hour, but right now it would really make a difference in our progress.
WDYT?
Dusty _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro...
On 8/20/19 10:18 AM, Clement Verna wrote:
On Tue, 20 Aug 2019 at 15:30, Dusty Mabe dusty@dustymabe.com wrote:
On 8/19/19 6:09 PM, Kevin Fenzi wrote:
Well, this I think is related to our "agile transformation" (or if you prefer "reorganizing how we work"). Right now, you find it hard to get stuff done because everyone is busy on other things, so you have to nag us to get stuff done which makes things worse for us, etc.
Right. It has been this way for a long time. It's not your fault and it's not my fault, but both of us feel like we've got the short end of the stick.
Ideally I think we can get to a place where we do a lot more scheduling and a lot less having to yell for cycles. You note a few things were just 5 min for someone to do... but the way we are working now someone does that thing for you finally, but then goes back to the other stuff they were working on, meaning in 30min you need another thing... repeat.
If you can come to us and say "hey, this is my project, I need it deployed by X" we can look at it, gather info we need, put it in our queue and then tell you "hey, we can work on that next tuesday". Then next tuesday we can devote a block of time to getting everything done. I think thats a win for everyone in the end and I would really like to get there. Can we? I sure hope so
In fact perhaps we could start doing this now somewhat: block off say... wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
I think blocking off time and doing some scheduling is a great idea. My advice here is that we have a shared calendar with blocks that people can sign up for automatically (i.e. if the block is open anyone can reserve it as long as they have a FAS account).
The only counter point that I have is that sometimes a "block of time" isn't always what's needed. Sometimes it will be something very small, but rather fundamental, that requires quite a bit of rework by the end user. For example, you and I have a 5 minutes session where some revelation comes to light and I need to go rewrite a portion of my application. I then spend two hours doing that and have to wait a week for the next block of time.
I think this is the symptom of a lack of upfront cooperation and design. This is what we are currently putting in place, if you need something consequent from us you come up with a spec that explain the WHAT and the WHY. Then we work together on the HOW and WHEN, once we have a good idea of the HOW and WHEN the work will be prioritized on our side (might have a dedicated team assigned to it) which I think will make cooperation easier.
It is never too late to start, so maybe we should spend some time to write down everything that is needed for Fedora CoreOS, write down HOW this will be done and what the work being Done means. That way we could probably dedicated a few people to focus on that work until completion.
How does that sounds ?
We certainly haven't done as good as we could here but we have written down our overall design [1] and had several sessions with core members of releng and infra (mohan, patrick, kevin) to discuss the feasibility of it all. Once we got to a point where we were ready to formally ask for help we have done that in tickets to the infra repo [2-6]:
We're definitely not doing the best job here, but we have done some communication. I do want to say that I'm not complaining about the job anyone has done! Just trying to find a better move forward for everyone.
Dusty
[1] https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.m... [2] https://pagure.io/fedora-infrastructure/issue/7884 [3] https://pagure.io/fedora-infrastructure/issue/7870 [4] https://pagure.io/fedora-infrastructure/issue/7821 [5] https://pagure.io/fedora-infrastructure/issue/8064 [6] https://pagure.io/fedora-infrastructure/issue/7997 [7] https://pagure.io/fedora-infrastructure/issue/7719
On Tue, Aug 20, 2019, 17:32 Dusty Mabe dusty@dustymabe.com wrote:
On 8/20/19 10:18 AM, Clement Verna wrote:
On Tue, 20 Aug 2019 at 15:30, Dusty Mabe dusty@dustymabe.com wrote:
On 8/19/19 6:09 PM, Kevin Fenzi wrote:
Well, this I think is related to our "agile transformation" (or if you prefer "reorganizing how we work"). Right now, you find it hard to get stuff done because everyone is busy on other things, so you have to nag us to get stuff done which makes things worse for us, etc.
Right. It has been this way for a long time. It's not your fault and
it's
not my fault, but both of us feel like we've got the short end of the
stick.
Ideally I think we can get to a place where we do a lot more scheduling and a lot less having to yell for cycles. You note a few things were just 5 min for someone to do... but the way we are working now someone does that thing for you finally, but then goes back to the other stuff they were working on, meaning in 30min you need another thing...
repeat.
If you can come to us and say "hey, this is my project, I need it deployed by X" we can look at it, gather info we need, put it in our queue and then tell you "hey, we can work on that next tuesday". Then next tuesday we can devote a block of time to getting everything done.
I
think thats a win for everyone in the end and I would really like to
get
there. Can we? I sure hope so
In fact perhaps we could start doing this now somewhat: block off
say...
wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
I think blocking off time and doing some scheduling is a great idea. My
advice
here is that we have a shared calendar with blocks that people can sign
up for
automatically (i.e. if the block is open anyone can reserve it as long
as they
have a FAS account).
The only counter point that I have is that sometimes a "block of time"
isn't
always what's needed. Sometimes it will be something very small, but
rather
fundamental, that requires quite a bit of rework by the end user. For
example,
you and I have a 5 minutes session where some revelation comes to light
and I
need to go rewrite a portion of my application. I then spend two hours
doing
that and have to wait a week for the next block of time.
I think this is the symptom of a lack of upfront cooperation and design. This is what we are currently putting in place, if you need something consequent from us you come up with a spec that explain the WHAT and the WHY. Then we work together on the HOW and WHEN, once we have a good idea of the HOW and WHEN the work will be prioritized on our side (might have a dedicated team assigned to it) which I think will make cooperation easier.
It is never too late to start, so maybe we should spend some time to write down everything that is needed for Fedora CoreOS, write down HOW this will be done and what the work being Done means. That way we could probably dedicated a few people to focus on that work until completion.
How does that sounds ?
We certainly haven't done as good as we could here but we have written down our overall design [1] and had several sessions with core members of releng and infra (mohan, patrick, kevin) to discuss the feasibility of it all. Once we got to a point where we were ready to formally ask for help we have done that in tickets to the infra repo [2-6]:
We're definitely not doing the best job here, but we have done some communication. I do want to say that I'm not complaining about the job anyone has done! Just trying to find a better move forward for everyone.
Yes just to make sure there is no misunderstanding I am not trying to blame anyone too and I am also keen on finding a better ways for these work to move smoothly without the need for you to bug someone every day and without having someone bugging us every day :-D.
I think in that particular case, the problem is that your needs went under our radar, and the work that you needed was not visible, hence not prioritized at our team level. This means that we have been busy with X other initiatives (Rawhide Gating, EPEL-8, Community OpenShift, + usual keeping the lights on work). Unfortunately it is almost impossible for you to see how busy we are and what are our current priorities and work in progress. We hope to improve in that area, so that I believe it would be easier for you to see what the team is busy with. Then it should be easier for you to raise your hand and say that you have something urgent coming in. In the same time we don't really have visibility to your planning and to your milestones, so it is also difficult for us to make a good call without knowing the expected timeline and what is your critical path.
All that to say I think the main problem here is lack of communication and lack of sharing information.
Dusty
[1] https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.m... [2] https://pagure.io/fedora-infrastructure/issue/7884 [3] https://pagure.io/fedora-infrastructure/issue/7870 [4] https://pagure.io/fedora-infrastructure/issue/7821 [5] https://pagure.io/fedora-infrastructure/issue/8064 [6] https://pagure.io/fedora-infrastructure/issue/7997 [7] https://pagure.io/fedora-infrastructure/issue/7719
On 8/20/19 11:52 AM, Clement Verna wrote:
Yes just to make sure there is no misunderstanding I am not trying to blame anyone too and I am also keen on finding a better ways for these work to move smoothly without the need for you to bug someone every day and without having someone bugging us every day :-D.
I think in that particular case, the problem is that your needs went under our radar, and the work that you needed was not visible, hence not prioritized at our team level. This means that we have been busy with X other initiatives (Rawhide Gating, EPEL-8, Community OpenShift, + usual keeping the lights on work). Unfortunately it is almost impossible for you to see how busy we are and what are our current priorities and work in progress. We hope to improve in that area, so that I believe it would be easier for you to see what the team is busy with. Then it should be easier for you to raise your hand and say that you have something urgent coming in. In the same time we don't really have visibility to your planning and to your milestones, so it is also difficult for us to make a good call without knowing the expected timeline and what is your critical path.
All that to say I think the main problem here is lack of communication and lack of sharing information.
We are totally in agreement. My request here was trying to short circuit the process since I know what our priorities are and you know what your priorities are we can make sure we sync up on the overall goals (i.e. high level sessions where we discuss what we want to do and how to do it, like the sesions we had with mohan, patrick, kevin before) and then someone from our team executes on those goals.
I understood from the beginning that it might not work. Just wanted to have the conversation.
Thanks for everything you do cverna! Dusty
On Tue, 20 Aug 2019 at 09:20, Dusty Mabe dusty@dustymabe.com wrote:
In fact perhaps we could start doing this now somewhat: block off say... wed into 2 hour blocks. Sign up people who have projects or pet bugs/issues they want to get solved for those blocks and work on them?
I think blocking off time and doing some scheduling is a great idea. My advice here is that we have a shared calendar with blocks that people can sign up for automatically (i.e. if the block is open anyone can reserve it as long as they have a FAS account).
The only counter point that I have is that sometimes a "block of time" isn't always what's needed. Sometimes it will be something very small, but rather fundamental, that requires quite a bit of rework by the end user. For example, you and I have a 5 minutes session where some revelation comes to light and I need to go rewrite a portion of my application. I then spend two hours doing that and have to wait a week for the next block of time.
That 5 minute session isn't a 5 minute session to us. For us it is usually 40-60 minutes of context swapping from what we were doing to fixing your 5 minute thing to try to get back to the other thing to going back to you.. but after someone sees we answered you.. they go and ping us expecting us to give them the same level of service we gave you. Then we never get back to the thing and you end up with us getting more and more frustrated by your 5 minute sessions. You are worried about waiting a week? Try waiting 2 years to complete a project because your ability to focus on it is constantly broken up by 5 minute sessions.
You having sysadmin-main is not going to fix this. It just means you are seen as responsible as everyone else who has sysadmin-main to fix every problem. You will now be getting the 5 minute pings that mizdebsk and the other non-Fedora Infrastructure people get all the time to fix all the 'little problems'. You will probably end up doing the same thing they do which is go off IRC completely for days at a time because it is the only way to get work done. You will also find that saying 'you can't help' doesn't get you anywhere.. we just get pinged asking why we have someone who can't do everything in sysadmin-main. [This has happened several times in the past with other people who have been in main so I am going from experience here.]
Please realize that you are one of multiple important customers who all want as much time from a sysadmin-main person as you do. We could triple the number of people in sysadmin-main and still not have enough time to deal with the irqs.. we would just find that a lot of them have just been paused indefinitely to spring up when more people are available. Slowing things down by more careful planning, processes and controls is usually the only way to deal with it because it is a human complexity problem.
I realize that sometimes this answer is not what works for a customer.
On 8/20/19 10:33 AM, Stephen John Smoogen wrote:
That 5 minute session isn't a 5 minute session to us. For us it is usually 40-60 minutes of context swapping from what we were doing to fixing your 5 minute thing to try to get back to the other thing to going back to you.. but after someone sees we answered you.. they go and ping us expecting us to give them the same level of service we gave you. Then we never get back to the thing and you end up with us getting more and more frustrated by your 5 minute sessions. You are worried about waiting a week? Try waiting 2 years to complete a project because your ability to focus on it is constantly broken up by 5 minute sessions.
You having sysadmin-main is not going to fix this. It just means you are seen as responsible as everyone else who has sysadmin-main to fix every problem. You will now be getting the 5 minute pings that mizdebsk and the other non-Fedora Infrastructure people get all the time to fix all the 'little problems'. You will probably end up doing the same thing they do which is go off IRC completely for days at a time because it is the only way to get work done. You will also find that saying 'you can't help' doesn't get you anywhere.. we just get pinged asking why we have someone who can't do everything in sysadmin-main. [This has happened several times in the past with other people who have been in main so I am going from experience here.]
Please realize that you are one of multiple important customers who all want as much time from a sysadmin-main person as you do. We could triple the number of people in sysadmin-main and still not have enough time to deal with the irqs.. we would just find that a lot of them have just been paused indefinitely to spring up when more people are available. Slowing things down by more careful planning, processes and controls is usually the only way to deal with it because it is a human complexity problem.
I realize that sometimes this answer is not what works for a customer.
I sympathize (empathize?) with everything you're saying smooge. I really hope this request isn't felt as being a pointed audit of anyone in sysadmin-main. I'm incredibly humbled to work with each of you every day. This request to me is more me saying "can I help by reducing the number of times we ping you?". But you're saying that what I'm asking for is different or actually won't help, which I accept.
Dusty
infrastructure@lists.fedoraproject.org
-
Clement Verna -
Dusty Mabe -
Kevin Fenzi -
Stephen John Smoogen