[Fiware-lab-recovery-tf] disruption in several idm based services

Álvaro Alonso aalonsog at dit.upm.es
Fri May 8 11:51:07 CEST 2015


Hi all, some updates regarding the current status:

- Yesterday we ran the script to activate the “privileged” permission to all the users. So they already have access to all the nodes. 

- I’ve just gave permissions to all IOs accounts in order to allow them to manage the accounts categories in the IdM portal. Federico is preparing a document with some tips explaining how to manage. Here only a reminder: we agreed yesterday that is important that a IO sends a reminder to a user whose community application has been rejected telling him that it will be a Trial user in a week.

- We are working right now on enabling an option in the IdM admin tab in order to be able to assign more that a region to a community user. It will be ready this morning.


BR!
-- 
Álvaro

> El 08 May 2015, a las 10:34, Juanjo Hierro <juanjose.hierro at telefonica.com> escribió:
> 
> 
>   I understand the matter.
> 
>   My experience is that this is only solved if exhaustive testing is made in advance.   The issue for that is that we don't have a proper testbed environment for that purpose.   This is something about which we should plan something next.   Nevertheless, my experience also is that nothing will allow you to avoid all issues in advance (because among other things, your testing environment does not reproduce all the casuistry (only real hard-critical systems can almost avoid that but it is because they have testing environments that are an exact reproduction of the real life environment) ... that's why in real production environments, updates are always perform in hours of little usage (i.e., nights) ... but that turns into the question about giving a service we didn't commit to give.  Either we change the SLAs we provide, or at least allow the ability to have these kind of night interventions (with the corresponding costs in terms of availability of personnal in extra hours, etc) or we will have to live with what we have.
> 
>   A continuous integration process may alleviate some problems but, IMHO, not so many in this particular respect (it does in others).
> 
>   I have been responsible of several business-critical systems in the pasts and know what has to be done, where the problems are and how they are prevented ... the issue is that we don't have a contract for providing such service (at least per the current contract) despite all what the EC may believe/desire.   We still have a research/pilot-like project contract.  Good news is that they may be willing to reconsider these aspects and may accept introducing changes in the budget and allocation of resources within FI-Core that grant providing at least more near business-critical system production operation standards.   Let's see how such discussion ends.
> 
>   Best regards,
> 
> -- Juanjo
> 
> On 08/05/15 10:06, Federico Michele Facca wrote:
>> Hi Juanjo,
>> Thanks, maybe what i was not able to highlight is what I wanted actually to say. I have the feeling that the IDM as such is not the problem, but that - for several reasons - we realize most of the issues only when we introduce things into production. We should avoid that, because this forces all of us to rush and get more stressed - because something is online changing a bit, and other things need to adapts.
>> 
>> The issue with putting in production the IDM could be the same for any other component indeed. Sure it is more evident because it is a central component. So maybe we miss a more coordinated sort of continuous integration process that could help us in foresee better such issues, I don't know, just an idea... but of course this requires more efforts... and if the people managing this are the ones that already try to solve 90% of the problem, we will indeed just create another problem, because honestly I don't think we can give more than this xD
>> 
>> Federico
>> 
>> 
>> On Fri, May 8, 2015 at 9:27 AM, Juanjo Hierro <juanjose.hierro at telefonica.com <mailto:juanjose.hierro at telefonica.com>> wrote:
>> Hi,
>> 
>>   Yesterday night we decided some actions that will help to overcome some issues users were going to experience.   In particular, the fact that there is a long queue of users who have applied to become Community Users but haven't being answered yet.   They had stopped seeing the nodes where they had their VMs and they would remain like that until their request is served.   If, for whatever reason, the coaches and the responsibles of nodes were not enough agile, there would be users who would stay days without seeing their VMs and not clue whether they have lost them forever.   This was not admisible.   You helped to identify this problem.
>> 
>>   We agreed to go for a quick&dirty solution: we allowed those users in the queue to see all the nodes.   They will become a short of "privileged" users while they are on the queue, but we took that decision because it is better that they see all the nodes than they don't their previous node just Spain-2.    Besides, we sent an email to all the users telling them that they may see all the nodes as the result of the migration but they shouldn't use any node different than the one they were already using.   We hope they will follow that recomendation.
>> 
>>   Then, we asked all coaches and responsible of nodes to quickly serve the queues of requests for becoming Community Users (please Alvaro or Fernando, confirm this is correct).     Once they serve the queue of pending requests, we should not have "privileged" users any longer ("privileged" users would turn into "Trial Users" because their application to upgrade was rejected, or "Community Users" because their application request was accepted).    Those "privileged" users who will remain once all requests have been served (these may be users who didn't formulate a request) would also be transformed into "Trial Users".    After this process, no "privileged" users would remain and question had been solved.
>> 
>>   Please Federico and Stefano, put pressure on coaches and responsible of nodes to serve the requests for upgrading user accounts as soon as possible.   This would help to approach the final target status as soon as possible.
>> 
>>   As per regard the issue about using types of grants for authorization which are not longer supported in the new version of the IdM GE, Alvaro sent an email to all users, notifying them what to do.   It may be a little unconvenient for some users (e.g., the developers of the infografic pannel) but we believe there won't be so many users affected.  UPM will try to implement support to the other two grant types as soon as possible, but we decided that it was acceptable asking users to change the type of grant they use.
>> 
>>   I hope I have been able to capture everything.   If Fernando or Alvaro need to amend something I have said, please do so.
>> 
>>   I believe we are doing overall fine, although we have always to try improving.   I also personally thank Fernando and Alvaro for their availability yesterday night to meet over the phone and implement the necessary actions to solve the issues we were facing.    
>> 
>>   Let's not get frustrated !  
>> 
>>   Best regards,
>> 
>> -- Juanjo
>> 
>> 
>> 
>> 
>> On 08/05/15 07:42, Federico Michele Facca wrote:
>>> Hi Alvaro,
>>> let me start saing that I do know you are doing your best, and honestly I believe the responsability is not yours: it lays on all of us.
>>> 
>>> We need to plan more careful and assert better actions, because of course you cannot know everything as other developers relying on the idm cannot as well. The frustration is not toward you, it's general in the feeling that we are always underestimating actions and their impacts and we always rush after fixing something. And we are all too busy with this to collaborate better on making things better.
>>> 
>>> that's all :(
>>> 
>>> Federico
>>> 
>>> 
>>> 
>>> On Thu, May 7, 2015 at 9:59 PM, Álvaro Alonso <aalonsog at dit.upm.es <mailto:aalonsog at dit.upm.es>> wrote:
>>> Hi, 
>>> 
>>> I’ve just sent an email to the users reminding them that everything works as before regarding oauth. Federico, honestly I didn’t know we support that gran type, we have implemented the two that we included in the documentation of the GE. So we thought that that was enough for be compatible. Anyway if you need them, we are going to include the support for the other ones. But sincerely, I’m sure too many users are using that because it is not explained as supported in our documentation. 
>>> 
>>> Regarding the issue with the community users, we have sent several massive notifications explaining the new situation with the accounts. I’ve also sent you an internal mail with the precise explanation so I thought it was so clear. I agree with you in the fact that users that sent the community application and have not been added as community users yet (even they are accepted) are not able to use the infraestructure in other regions than spain2 until we add them. During the day of today I had not a minute to process them. All I can do is start tomorrow morning explaining IOs how to proceed with the insertion of the approved accounts and even help you processing some of them. 
>>> 
>>> Believe me, we are doing our best for this.
>>> 
>>> Hope this helps. 
>>> 
>>> BR
>>> -- 
>>> Álvaro
>>> 
>>>> El 07 May 2015, a las 21:09, JUAN JOSE HIERRO SUREDA <juanjose.hierro at telefonica.com <mailto:juanjose.hierro at telefonica.com>> escribió:
>>>> 
>>>> 
>>>> Gentlemen ... I can hardly believe that we are going through this ... How was the migration procedure tested?  One of the reasons why we delayed deployment of the IdM until May was to allow proper testing of the new IdM and the migration procedure ...
>>>> 
>>>> Sorry but this requires immediate repair action ...  I know that we are not commiting a 24/7 SLA, but when so a big issue like this occurs, we have to go the extra mile ...
>>>> 
>>>> Alvaro?  Could you tell us where we are at the moment?
>>>> 
>>>> Sincerely speaking, we won't be able to afford one complete day out of service ... All the credit we had been able to recover with our hard work this last month will fly away ... I hope we all understand that ...
>>>> 
>>>> I stay tunned tonight waiting to hear about progress ...
>>>> 
>>>> Enviado desde mi iPad
>>>> 
>>>> Inicio del mensaje reenviado:
>>>> 
>>>>> De: Federico Michele Facca <federico.facca at create-net.org <mailto:federico.facca at create-net.org>>
>>>>> Fecha: 7 de mayo de 2015 20:34:37 GMT+02:00
>>>>> Para: Juanjo Hierro <juanjose.hierro at telefonica.com <mailto:juanjose.hierro at telefonica.com>>
>>>>> Cc: fernando <fernando.lopezaguilar at telefonica.com <mailto:fernando.lopezaguilar at telefonica.com>>
>>>>> Asunto: disruption in several idm based services
>>>>> 
>>>>> dear juanjo,
>>>>> there are number of services that are not working anymore, because the implemenation of oauth2 from the new idm and the supported grant are different.
>>>>> also we haven't informed users that they will need to wait that someone apply the status to them and assign to a node in the idm... (i understand alvaro, proposing the infrastructure to take care last night, since it is a quite big load, but then we should have warned users and infrastructure to be available today to fix that...) so we will get flooded with tickets (already actually) xD
>>>>> 
>>>>> i didn't suppose things going like this... honestly i am quite frustrated :(
>>>>> 
>>>>> tomorrow morning, i will ask the guys to have a look, but no gurantee infographics and term and conditions will get back working shortly.
>>>>> 
>>>>> federico
>>>>> -- 
>>>>> --
>>>>> Future Internet is closer than you think!
>>>>> http://www.fiware.org <http://www.fiware.org/>
>>>>> 
>>>>> Official Mirantis partner for OpenStack Training
>>>>> https://www.create-net.org/community/openstack-training <https://www.create-net.org/community/openstack-training>
>>>>> 
>>>>> -- 
>>>>> Dr. Federico M. Facca
>>>>> 
>>>>> CREATE-NET
>>>>> Via alla Cascata 56/D
>>>>> 38123 Povo Trento (Italy)
>>>>> 
>>>>> P  +39 0461 312471 <tel:%2B39%200461%20312471>
>>>>> M +39 334 6049758 <tel:%2B39%20334%206049758>
>>>>> E  federico.facca at create-net.org <mailto:federico.facca at create-net.org>
>>>>> T @chicco785
>>>>> W  www.create-net.org <http://www.create-net.org/>
>>>> 
>>>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
>>>> 
>>>> The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
>>>> 
>>>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
>>>> _______________________________________________
>>>> Fiware-lab-recovery-tf mailing list
>>>> Fiware-lab-recovery-tf at lists.fiware.org <mailto:Fiware-lab-recovery-tf at lists.fiware.org>
>>>> https://lists.fiware.org/listinfo/fiware-lab-recovery-tf <https://lists.fiware.org/listinfo/fiware-lab-recovery-tf>
>>> 
>>> 
>>> _______________________________________________
>>> Fiware-lab-recovery-tf mailing list
>>> Fiware-lab-recovery-tf at lists.fiware.org <mailto:Fiware-lab-recovery-tf at lists.fiware.org>
>>> https://lists.fiware.org/listinfo/fiware-lab-recovery-tf <https://lists.fiware.org/listinfo/fiware-lab-recovery-tf>
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> --
>>> Future Internet is closer than you think!
>>> http://www.fiware.org <http://www.fiware.org/>
>>> 
>>> Official Mirantis partner for OpenStack Training
>>> https://www.create-net.org/community/openstack-training <https://www.create-net.org/community/openstack-training>
>>> 
>>> -- 
>>> Dr. Federico M. Facca
>>> 
>>> CREATE-NET
>>> Via alla Cascata 56/D
>>> 38123 Povo Trento (Italy)
>>> 
>>> P  +39 0461 312471 <tel:%2B39%200461%20312471>
>>> M +39 334 6049758 <tel:%2B39%20334%206049758>
>>> E  federico.facca at create-net.org <mailto:federico.facca at create-net.org>
>>> T @chicco785
>>> W  www.create-net.org <http://www.create-net.org/>
>> -- 
>> 
>> ______________________________________________________
>> 
>> Coordinator and Chief Architect, FIWARE platform
>> CTO Industrial IoT, Telefónica
>> 
>> email: juanjose.hierro at telefonica.com <mailto:juanjose.hierro at telefonica.com>
>> twitter: @JuanjoHierro
>> 
>> You can follow FIWARE at:
>>   website:  http://www.fiware.org <http://www.fiware.org/>
>>   twitter:  @FIWARE
>>   facebook: http://www.facebook.com/pages/FI-WARE/251366491587242 <http://www.facebook.com/pages/FI-WARE/251366491587242>
>>   linkedIn: http://www.linkedin.com/groups/FIWARE-4239932 <http://www.linkedin.com/groups/FIWARE-4239932>
>> 
>> 
>> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
>> 
>> The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
>> 
>> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição
>> 
>> 
>> 
>> -- 
>> --
>> Future Internet is closer than you think!
>> http://www.fiware.org <http://www.fiware.org/>
>> 
>> Official Mirantis partner for OpenStack Training
>> https://www.create-net.org/community/openstack-training <https://www.create-net.org/community/openstack-training>
>> 
>> -- 
>> Dr. Federico M. Facca
>> 
>> CREATE-NET
>> Via alla Cascata 56/D
>> 38123 Povo Trento (Italy)
>> 
>> P  +39 0461 312471
>> M +39 334 6049758
>> E  federico.facca at create-net.org <mailto:federico.facca at create-net.org>
>> T @chicco785
>> W  www.create-net.org <http://www.create-net.org/>
> -- 
> 
> ______________________________________________________
> 
> Coordinator and Chief Architect, FIWARE platform
> CTO Industrial IoT, Telefónica
> 
> email: juanjose.hierro at telefonica.com <mailto:juanjose.hierro at telefonica.com>
> twitter: @JuanjoHierro
> 
> You can follow FIWARE at:
>   website:  http://www.fiware.org <http://www.fiware.org/>
>   twitter:  @FIWARE
>   facebook: http://www.facebook.com/pages/FI-WARE/251366491587242 <http://www.facebook.com/pages/FI-WARE/251366491587242>
>   linkedIn: http://www.linkedin.com/groups/FIWARE-4239932 <http://www.linkedin.com/groups/FIWARE-4239932>
> 
> 
> Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.
> 
> The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.
> 
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fiware.org/private/fiware-lab-recovery-tf/attachments/20150508/bdfce088/attachment.html>


More information about the Fiware-lab-recovery-tf mailing list

You can get more information about our cookies and privacy policies clicking on the following links: Privacy policy   Cookies policy