Phases complete and shown as successful even if a task fails

[wagnerand]

Recently, I triggered the "Ship" phase for firefox-translations-1.1.2-build which completed according to shipit:

However, looking at the task group in task cluster, one task has failed:

---

Similarly, the a build for searchengine-devtools failed (https://firefox-ci-tc.services.mozilla.com/tasks/groups/SzR0NU1xThGCnCKdi2Datw), but shipit showed completed and even allowed me to sign off on it. I have canceled that build in the meantime. The same goes for a recent remote-settings-devtools build

In the task-group, the linter task failed:

[escapewindow]

This is the current behavior: relman, sheriffs, and releng watch the graphs to make sure they go green. So one way would be to require addons reviews/signoffs to include a green task group check.

We could add a shipit task in each phase that marks the phase as completed, but a) there are some cases where non-blocking leaf nodes fail in the Firefox release graph and we decide that that's ok, so we can't make it a hard failure; and b) we'll have to slot that into Releng's current priorities.

[wagnerand]

Thank you. I believe blocking would be ideal for add-ons. If that is not feasible or takes a long time, could we start by non-blocking, but showing something other than the blue checkmark on shipit? This would help users to immediately see if something requires their attention, without having to (remember to) inspect the tasks every time.

[gabrielbusta]

I started taking a stab at this yesterday. I think both are feasible. I'd need to make changes to keep track of the release phase status.

• The shipit back-end changes to add a phase status API might look like this
• Calling a phase status API from shipitscript might look like this

[gabrielbusta]

#1087 makes is it so the ShipIt phase progress accounts for the state of tasks in the phase's task group. Making it non-blocking so Firefox releases can still be signed off if there's some failing tasks. We can also block on failed tasks for add-ons and allow sign offs with failed task group tasks for Firefox.