Staging Workflow
The Open Build Service (OBS) is well known for providing an easy way to build and distribute binary packages from source code. The packages, grouped together in what we call a project, are built every time they are updated. The maintainer of the package can choose among a wide range of operating systems and hardware architectures to build the packages on. Those continuous building processes ensure that the packages are always working for the different setups.
The maintenance of those packages can be made on a collaborative way via OBS. As shown in the following diagram, the maintainer can create a package and then he or any other developer can branch it (make a copy of it), can do some changes on its code and can request those changes to be applied on the original package. After that, the maintainer usually reviews the request, chats with the developer in case it needs some fixes and ends up accepting the request. Doing so, the new changes to the code become part of the package's source code.
However, the workflow is not always that easy. Apart from managing individual packages, OBS provides many other functionalities and it even allows us to release entire distributions. In a very simplistic way, we can say a distribution is just an OBS project with thousands of packages inside. Packages that have been selected to be installed together as part of the distribution.
When dealing with such a big project, many people request changes in many different packages all the time. They have to be reviewed, adjusted and tested (built) before being accepted. As you can imagine, it becomes nonviable to review the packages one by one. Even if the maintainer checks that a package is not broken and merges it, it can break everything else for conflicts with other packages. To deal with these situations, OBS provides what we call Staging Workflow.
The idea behind the Staging Workflow is testing the requests incrementally by batches. First, a copy of the original project is created, it is called Staging Project and is going to act as a playground. The Staging Manager selects some of the requests he considers they belong together and assigns the corresponding packages to the Staging Project. This way, the groups of packages are going to be tested (built) in one go. Once the Staging Project gets built, the changes can be merged to the original project.
The Staging Manager can create as many Staging Projects as he considers and can assign different selections of requests to each of them. It is still tedious solving the conflicts that appear between them, but being able to test a lot of packages in parallel is much more efficient than doing the same package by package.
Let's make it clearer with a real example. Imagine we are working on the project openSUSE Factory and start working on its Staging Workflow.
Many contributors and maintainers really want some improvements to be applied on their packages, so the openSUSE:Factory project receives new requests all the time. Among all of them, there are a few that are related to Gnome packages, so the Staging Manager decides to stage them together in openSUSE:Factory:Staging:A. The Staging Project is an exact copy of the main project openSUSE:Factory.
The building process begins and, if something gets broken, the Staging Manager asks the requestor to fix it. This can add even more requests to the scene but the goal is always getting a working version of openSUSE:Factory:Staging:A by fixing or even discarding some of the requests. When the building process finishes successfully, the requested changes are merged in the source code of openSUSE:Factory and some other batches of requests are staged again and again until we come up with an stable version of openSUSE:Factory ready to be released.
- A Project has one StagingWorkflow
- A StagingWorkflow has many StagingProjects
- A StagingProject has many StageRequests
- A StagedRequest has one BsRequest
- A BsRequest has many Reviews
Anyone can setup a Staging Workflow in their own project. There are to ways to do it: using the web Interface or the osc tool.
If you use the web interface, you just need to go to the Staging tab in the project page, assign the "Managers Group" which will be the owner of that Staging Workflow and click on "Create Staging Projects".
Only the owner of the project and the users of the chosen Managers Group can manage the Staging Workflow.
To do the same thing using the osc tool, you only have to run the next command, specifying the group that will manage the Staging Workflow:
osc api -X POST '/staging/openSUSE:Leap:15.2/workflow' -d '<workflow managers="some_group"/>'
Once the Staging Workflow is created successfully, two empty Staging Projects will be ready for you. Following the example, if the project name is openSUSE:Leap:15.2, they'll be named after it like openSUSE:Leap:15.2:A and openSUSE:Leap:15.2:B.
The purpose of those projects is to work as a playground area to test several packages together. Of course you can create as many extra Staging Projects as you need or assign an existing Staging Project to a Staging Workflow where you have rights (e.g: a subproject).
As any other project, the staging project could have a specific configuration (repositories, users and groups, etc) therefore, sometimes it can be useful to copy an existing staging project (template), instead of creating one from scratch.
For example, if we use openSUSE::Leap:15.2:Staging:A as template, the new staging project will contain the same configuration, repositories, users and groups than the original one.
osc api -X POST '/staging/openSUSE:Leap:15.2/staging_projects/openSUSE:Leap:15.2:Staging:A/copy/openSUSE:Leap:15.2:Staging:A_copy'
With the initial setup done, we can start using the Staging Workflow.
Create a branch from a package you intend to contribute to. Go to the package and click on the "Branch Package" button. Now you can add the changes you need to the files of the branched package.
After modifying the branched package, is time to submit a Request to ask for a review on the changes.
A request in 'review' state is created and shown in the Backlog section of the Staging Workflow page. That Request will have a Review for the managers group, what we call a Review by_group.
At this point, the Staging Project is in acceptable state.
The main purpose of the Staging Workflow is to provide a way to stage a bunch of packages and ensure that they work well together.
To stage the requests together we have to use the osc tool:
osc api -X POST '/staging/openSUSE:Leap:15.2/staging_projects/openSUSE:Leap:15.2:Staging:A/staged_requests' -d '<requests><request id="705329"/><request id="705365"/></requests>'
After this we could check if it was correctly staged with the following command, that will show to us all the requests that are staged in a staging project.
osc api '/staging/openSUSE:Leap:15.2/staging_projects/openSUSE:Leap:15.2:Staging:A/staged_requests'
Once the Request is assigned, it gets another Review against the staging project provided before. This review its a Review by_project.
The Request disappears from the Backlog and is shown next to the Staging Project name in the table.
When testing some packages together in a Staging Project, we might get conflicts or realize that any of them is no longer valid.
In such cases, we can unstage a Request and test the Staging Project again with the rest of the packages.
The unstaged requests will be sent back to the Backlog.
osc api -X DELETE '/staging/openSUSE:Leap:15.2/staged_requests' -d '<requests><request id="705365"/></requests>'
Unfortunately, at the moment moving staged request between Staging Projects is not possible, the only way is unstaging the request and staging it into a different Staging Project.
Sometimes it can be useful to exclude a request and don't let it be available in the Backlog.
This can prevent the staging project from being assigned with requests we are sure are causing conflicts, have some missing dependencies or have to wait for other request to be accepted.
It is always important to describe the reason to exclude a request.
We can exclude requests via the Web UI one by one, going to the Staging Overview page, clicking on the Excluded link in the Infos section, then clicking on the "Exclude requests" link at the bottom of the page.
We can do it also via the osc tool, and we can exclude several requests at once:
osc api -X POST '/staging/openSUSE:Leap:15.2/excluded_requests' -d "<excluded_requests><request id='670595' description='Lorem ipsum A'/><request id='670596' description='Lorem ipsum B'/></excluded_requests>"
If we stop having reasons to exclude a request, we can remove the exclusion and the request will return to the Backlog, ready to be staged into a Staging Project.
osc api -X DELETE '/staging/openSUSE:Leap:15.2/excluded_requests' -d '<excluded_requests><request id="670595"/></excluded_requests>'
Once all requests are ready and the staging project has an acceptable state, we can merge (accept) the Staging Project.
In other words, all the requested changes we tested in the Staging Project can now be integrated into the main project, in this case openSUSE:Leap:15.2.
To accept/merge the staging project we have to use the following command from osc:
osc api -X POST '/staging/openSUSE:Leap:15.2/staging_projects/openSUSE:Leap:15.2:Staging:A/accept'
Given a Staging Workflow project we can define several Required Checks and associate them to:
- Requests: to check sources.
- Repository or Repository+Architecture: to check binaries.
Once they are built and published.
When the checks finish, they will report back to OBS with the check result, and the final status of the checks will be available here:
http://example.com/staging_workflows/3/staging_projects/openSUSE:Leap:15.2:Staging:A
How the tools start checking whatever they have to check, is up to the tool to decide. We only recommend to channel all communication with OBS via the message bus.
If all the required checks succeed the Staging Project can be accepted, and they will be merged onto the main project.
Otherwise, they should be reviewed to have the staging project in an acceptable state.
https://build.opensuse.org/apidocs/index#177
Requests with open review by group that are not staged or excluded.
To tell people that the request is going to be staged, and to prevent this request to be merged into the main project.
Requests that the manager doesn't want to include in a stage. The belong to the model Staging::RequestExclusion.
The overall_state method defines whether a StagingProject can be accepted or not. The following conditions should be met to make a StagingProject acceptable:
- if it is not empty
- if there are no untracked requests or obsolete requests (requests in state 'declined', 'superseded' or 'revoked')
- if there are no unfinished builds for any StagedRequest of a StagingProject
- if there are no open reviews for any StagedRequest of a StagingProject
- if there are no open StatusChecks for any Repositories of a StagingProject
Accepting a StagingProject means to accept all requests that are staged for this project. Once the requests have been accepted, they are unstaged from the project. While a StagingProject is being accepted it is in state 'accepting'. Once this is done it's state is 'empty'.
@staging_project.staged_requests.each do |staged_request|
staged_request.request.reviews.accept # the by project review
staged_request.request.accept....
staged_request.destroy # Remove the relationship
endStagedRequest, is a request which is assigned to a specific StagingProject
StageRequest is ready if it doesn't have any open reviews.¹ A StageRequest cannot be excluded once it's "ready".
¹ The review by project on the StagingProject is ignored. See the question below.
This prevents users from merging BsRequests that are staged in a StagingProject and visualizes which StagingProject the request is staged to. Be aware that technically it would still be possible to accept such a BsRequest.
StageRequest is "In review" if it has at least one open review.¹
StageRequest is "Obsolete" because it is in the state: declined, superseded or revoked.
Shows if the BsRequest has a BsRequestActionDelete action for deleting a package
The idea is to show if there is any build problems, or status checks, blocking the acceptance of the StagingProject.
It shows the progress of the current state, for example 2 requests with 7 open reviews and 3 closed ones would have a review progress of 70%.
The official documentation:
- https://openbuildservice.org/help/manuals/obs-best-practices/cha.obs.best-practices.webuiusage.html#staging_how_to
- https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.stagingworkflow.html
Customer needs on Story map:
Wiki:
- https://github.com/openSUSE/open-build-service/wiki/Staging-Workflow
- https://github.com/openSUSE/open-build-service/wiki/Staging-Workflow-Step-by-Step
- https://gist.github.com/saraycp/ddda1cc4b622149926287b0cadede46b
OBS documentation:
- https://openbuildservice.org/help/manuals/obs-best-practices/cha.obs.best-practices.webuiusage.html#staging_how_to
- https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.stagingworkflow.html
Maintenance Setup Example: