# Welcome to ECSE 437 (Fall 2023)!
We build something beyond code! we build friendships!
{% hint style="danger" %}
**Important Notice: Outdated Course Material**
\
Please be aware that the course materials you are accessing were designed for students who took this course in Fall 2023 with [Prof. Majid Babaei ](https://www.mcgill.ca/continuingstudies/majid-babaei)at McGill University.
If you have any questions or concerns about the relevance of the material, please don't hesitate to reach out to your current instructor or the course coordinator.
{% endhint %}
{% hint style="info" %}
**About this page:** This page is designed as supplementary material to ECSE 437: Software Delivery offered by McGill University in Fall 2023. Get access to the official content in myCources at [ECSE-437-001](https://mycourses2.mcgill.ca/d2l/home/659807).
{% endhint %}
{% hint style="success" %}
To keep you updated about the most recent changes in this course, a [news](/news) section has just been added!
{% endhint %}
Before jumping to the more serious stuff about the course let me bring your attention to some important points:
How to use it?
This space is designed to be read linearly, so start with the Team Section and work down from there! Some sections such as **Lecture Notes** are still in progress and will not be used as resources in any exams.
Contributing
If you want to contribute changes, start a new change request and submit it for review. We use the [fork and pull](https://help.github.com/articles/using-pull-requests/#fork--pull) model to manage changes. More information about [forking a repository](https://help.github.com/articles/fork-a-repo/) and [making a Pull Request](https://help.github.com/articles/using-pull-requests/).\
***If your pull request is accepted you will be awarded with 1 to 5 bunes points!** (typos are not included!)*
# News
🥼 Labs
### No Sections!
There are no sections in the lab sessions! you can join either session or both sessions each week. But I'd strongly recommend you to join both sessions to be able to prepare a fantastic portfolio for your team that will come in handy after this course or even after you graduate from McGill :tada:
***
### Attendance is not obligatory!
Your *physical* presence in the lab sessions is not mandatory! But labs cannot be done individually! it would be best if you worked in a team.
***
### Fuel for your discussion during the tutorial sessions \[How to write my Learning Journal?]
Dear Students,
If you have never written a Learning Journal in your other courses, it might seem a bit tricky! but actually, it is not! It is a practice to help you build your critical thinking which is essential in your future steps.
In the template I have provided for you, there are a few sections:
\[Questions] As you are making progress in the lab you might face a few questions. For example: *How many tasks should be assigned to a person in a sprint? Is there any optimal value or it depends on the nature of the tasks? Are there any relevant studies on the Internet to determine this value? How is this managed by large companies?*
You can either try to answer these questions on your own (based on your previous experiences) or you might want to do some research on the Internet. If you have found any reasonable answer to the questions you have faced you can add them with your explanation into the proper part of the learning journal which is \[Description of our approach].
I understand some questions are not easy to answer and require a lot of technical expertise. But it is not always the case! there are some easy solutions even for seemingly difficult problems! If the solutions that you have found for your questions sound interesting to you, you can try them in your project and see what are the results. For example, for this question: *You have a large task that needs to be completed in a sprint. How this task can be broken down into smaller pieces? ,* you might want to explore and try different options in your project and discuss it with people on your team. Then you can add your contributions int the \[our contribution] part of the learning journal.
In future labs, we will have different concerns but you should take the same steps to complete your learning journals!
🎁 Presentations
### Topic 10 is taken!
[Topic#10](/presentations/requesting-time-off-3/topic-10) is taken by Abraham Somech! We are very looking forward to your presentation on the last day of this course! -> *Liam Serour and Samuel Vasserman were added to the team.*
***
### Topic 2 is taken!
[Topic#2](/presentations/requesting-time-off/topic-2-taken) is taken by Felicia Sun! Visualization Support is a crucial technique that has been used widely in many domains. I am sure that you will enjoy this presentation so stay tuned for more information!
***
### Topic 3 is taken!
[Topic#3](/presentations/requesting-time-off-1/topic-3) is taken by Zachary Hayden! Code Review plays a critical role in software development and can be applied with the help of many tools and techniques. In this presentation, we will learn how this is managed at Google!
***
### Topic 9 is taken!
[Topic#9](/presentations/requesting-time-off-3/topic-9) is taken by Biruk berhanu Retta! what is more important than testing is software engineering and creating bug-free software especially if it is powered by AI!
***
### Topic 4 is taken!
[Topic#4](/presentations/requesting-time-off-1/topic-4-taken) is taken by Alexa Vasilakos! yet another topic on code review that shows us some hidden challenges in this part of software development.
### Topic 1 is taken!
[Topic#1](/presentations/requesting-time-off/topic-1-taken) is taken by Soumaia Bouhouia! Mining git repositories is a topic that has recently attracted great attention and several amazing papers have been published on this topic. In this presentation, we will learn the promises and perils of mining git.
***
### Topic 7 is taken!
[Topic#7](/presentations/requesting-time-off-2/topic-7) is taken by Marie Nashed! -> *assigned to Hadi Ghaddar and Omar Marwan*
***
### Topic 5 is taken!
[Topic#5](/presentations/requesting-time-off-2/topic-5-taken) is taken by Sandy Nguyen, Minna Feng, and Béatrice Duval!
***
### Topic 6 is taken!
[Topic#6](/presentations/requesting-time-off-2/topic-6-taken) is taken by Arman Shroff-Mehrabadi!
***
### Topic 8 is taken!
[Topic#8](/presentations/requesting-time-off-3/topic-8-taken) is taken by Ningning Yang!
***
### Summary of Topic 2 added
Felicia Sun successfully created a PR and after getting the approval the summary of the paper was included in [Topic#2](/presentations/requesting-time-off/topic-2-taken).
***
🍨 Quizzes
### Flexibility for Quiz1
I have recently been informed that the department student society (ECSESS) is organizing an industry trip to Toronto which leaves in the early morning on the 21st, on the same date as Quiz1! To accommodate students who are joining this event I can distribute the grade of this quiz on other quizzes evenly! **If you are going to join this session, please let me know as soon as possible!**
***
🎉 Final Project
### Best Teams:
Thanks for all you have done in this course and providing beautiful protfolio for your team during the lab assimgnets and the final project.
Among all the projects that I have reviewed the following teams have shown the best performance in terms of the clarify of the presentation, providing supplementary material and the end result:
1. Team 7: Sarah Youinou and Samer Sawan
2. Team 4: Afnan Waheed Ahmed, Sia Ham, and Nick Yoo
3. Team 20: Tara Ginsberg, Liam Serour, and Samuel Vasserman
4. Team 22: Vivek Kandathil, Richard Rassokhine, and Alexa Vasilakos
5. Team 24: Joey Liu Liong Wah, Qinghan Zhang, and Eric Zhang
**They will recieve 5 bonus points for their significant contribution in the final project.**
***
### Final project deliverables:
For the final project, each team needs to put all the steps they have taken in lab assignments and create a complete DevOps workflow for their team portfolio project. Each team should provide the following output:
* A link to your deployed project using the webApp component in the Azure DevOps environment.
* Complete the Team Portfolio - Final Project Template.
* Create a 20-minute video presentation of your project in which everyone explains what they did in the project and how the project was managed using DevOps methodology (you can upload the video to your OneDrive account and share it with me)
\
**Note 1**: You don't need to create a new project in your Azure DevOps profile. You can use the existing one.
**Note 2**: The deadline to submit your package is December 4th (11:59 pm)
**Note 3**: Make sure you share your video presentation with me otherwise I will not be able to view your file.
**Note4**: To complete Team Portfolio - Final Project Template: in the first part "OUR CONTRIBUTION" you should mention the main contribution you have made to each lab; in the second part "CHALLENGE#1, #2, #3" you only need to mention 3 main challenges you have faced throughout this project; in the third part "CONCEPTUAL DESIGN" you should draw your build and release pipeline you considered in your DevOps system; in the final part "WHAT ARE MISSING" you should talk about the missing part of the project, e.g., testing, and explain how this step can be integrated into your project.
***
🤶 Others
### Have you joined ECSE 437 recently?
First of all welcome to ECSE 437 I hope you enjoy this class and start building a solid foundation for your future steps!
In this class, you are supposed to work on a project called team-portfolio in which you collaborate with (mostly) two other students to implement a DevOps pipeline for your project. Here are some important points that are worth mentioning:
* If you don't have a team please contact Sarvin (), she will assign you to a team.
* Attendance is not mandatory for the tutorial sessions! you can join one or two sessions depending on the time you want to dedicate to your project.
* In each lab, you need to complete a learning journal that basically shows what you have learned in the lab, what challenges you have faced, and how you managed to resolve them.
* The final project will be putting all the steps you have taken in the labs sessions, creating a 20-minute presentation, and explaining your main contributions in your report.
* If you want to earn 10 bonus points you have two options: 1) Booking one slot presentation from the available topics; 2) Creating an excellent learning journal and demonstrating significant contributions in each lab (this will be evaluated by the TA)
* All quizzes contain short answer questions and will be held at the beginning of the class. So, please don't be late!
***
# Vision and Mission
## Our Vision
The Faculty of Engineering's vision at McGill University is to prepare globally-minded leaders who are equipped to solve the problems that matter.
We aim to deliver the highest quality of engineering education through teaching and research excellence, a state-of-the-art educational environment, and innovative programs.
## Objectives and strategies
* Instil life-long learning abilities and prepare professionally competent and broadly educated students capable of addressing the demands of the twenty-first century.
* Conduct high-caliber basic and applied research in engineering, applied science, and related fields.
* Promote a vibrant and fulfilling student-centered experience and foster student success by providing necessary support.
* Utilize advanced and innovative teaching and learning methods and tools, including advanced information, computing, and communication technologies, to facilitate successful learning within a technology-enriched learning environment.
* Foster a collegial, respectful, and productive environment for students and create a sense of spirit and loyalty.
* Engage in value-added activities that serve and address the needs of industry and engineering professions, and advance and improve the economic, environmental, and social welfare of the region, province, and country.
\
# Values
## Our Values
🚀 Integrity
As an undergraduate student, you will have many assignments, papers, laboratory reports and/or presentations to prepare over the course of your studies. You may also become involved in peer or public education, go on to do graduate studies, or become a teaching assistant (TA) for undergraduate courses.
As such, academic integrity may apply to you not only as a student, but potentially in these other roles as well. During your time at McGill, you may encounter the following scenarios that raise questions about academic integrity. For each, we indicate if it involves a violation of the *Code of Student Conduct and Disciplinary Procedures (the Code)*, the specific article of the Code that is violated, an explanation of what constitutes a particular offence, how a TA or instructor can prevent similar situations from occurring and the immediate consequences. The penalties for a violation of the Code are listed in the [Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/policies-student-rights-and-responsibilities). **Take the time to review the range of possible sanctions for a violation of the Code.**
✊ Honesty
Academic integrity is of vital importance to any serious educational institution. The art of scholarship demands a rigid insistence on giving credit where credit is due, and any failure to do so undermines not only the value of honest students' work, but also the academic integrity of the University and the value of a McGill degree.
This site provides [resources](https://www.mcgill.ca/students/srr/publications/) that can help students [avoid dishonest work](https://www.mcgill.ca/students/srr/honest/students/test) and the [disciplinary measures](https://www.mcgill.ca/students/srr/honest/students) that go with it, as well as useful information for [teaching staff](https://www.mcgill.ca/students/srr/honest/staff/).
💪 Equity, Diversity & Inclusion
Universities across Canada are presently called upon to recognize and address historical and contemporary forces that result in social inequities in postsecondary contexts. Many such forces have their roots in ideologies and practices – such as colonialism, slavery, and patriarchy. Although these ideologies and practices no longer reflect McGill’s values, their harmful effects persist. As such, our institutional commitment to equity, diversity, and inclusion (EDI) must acknowledge and seek to address the lasting effects of historic injustices that continue to challenge equal opportunities to access, and to succeed within, the McGill community. Our EDI commitments must also be inspired by the recognition that excellence is fostered by bringing together individuals and groups of diverse experiences, identities, and ideas. This Strategic EDI Plan for McGill seeks to act on this commitment through the articulation of specific goals, and measures for their achievement, over the next five years. Over this period, McGill will embed EDI in all core areas of the University, drawing on multiple strategic University-level documents initiated by McGill’s Principal and Provost over the last decade.
# Meet the Team!
Talent wins games, but teamwork wins championships!
{% hint style="info" %}
In this course, we can tap into incredible resources and wonderful people!
{% endhint %}
## Dr Majid Babaei
👋 Instructor — \
Personal website: [www.majidbabaei.com](http://www.majidbabaei.com/)\
Office: 680 Sherbrooke Street West, 13th floor, room#1324\
School of Continuing Studies.
{% hint style="info" %}
Online live Sessions, over Microsoft Teams, are held on Fridays 3:30 pm (by appointment). These sessions, usually, run for 30 minutes. However, they may also run for less than that in case there are no further questions or discussions.\
Lecture slides will be posted over MyCourses. If you have questions and need clarifications about the lecture slides, then the live sessions will be the perfect location to get your questions answered and clarified.
{% endhint %}
## Sarvin Ghiasi Khalehoghli
👋 TA —
Sarvin will be responsible for running tutorial sessions on Mondays and Wednesdays (1:35 p.m. - 2:25 p.m.) at RPHYS 118.
{% hint style="info" %}
If you run into any issues in doing your assignments in the tutorial sessions Sarvin is your best friend who is always eager to assist you.
{% endhint %}
## Motahareh Pour Ahimi
👋 TA —
Motahareh will be helping me with marking lab assignments, quizzes, and the final exam.
{% hint style="info" %}
If you have questions about your mark or want to make sure you have been treated fairly Motahareh is someone you can trust!
{% endhint %}
# Textbook
{% hint style="info" %}
Your first place to search for your favorite book title is [McGill Library](https://www.mcgill.ca/library/)!
{% endhint %}
### Required Textbooks
There is no required textbook for this course! However, I will be using a series of resources to teach this course.
### Optional Textbooks
1. [Achieving DevOps: a novel about delivering the best of Agile, DevOps, and microservices](https://mcgill.on.worldcat.org/oclc/1102581230)
2. [Learning DevOps](https://mcgill.on.worldcat.org/oclc/1308983414)
3. [CODE REVIEW GUIDE - OWASP Foundation](https://owasp.org/www-pdf-archive/OWASP_Code_Review_Guide_v2.pdf)
4. [Implementing effective code reviews: How to build and maintain clean code](https://mcgill.on.worldcat.org/oclc/1195455840)
5. [Software build systems: principles and experience](https://mcgill.on.worldcat.org/oclc/718419490)
6. [Static Program Analysis](https://cs.au.dk/~amoeller/spa/spa.pdf)
7. [Introduction to software testing](https://mcgill.on.worldcat.org/oclc/180851905)
8. [Continuous Delivery](https://martinfowler.com/books/continuousDelivery.html)
# Course Schedule
This page will be updated if any of deadlines is extended!
#### Dates, Topics, and assignments:
No
Date
Topics
Labs
Presentations
1
Aug. 31st
Introduction
2
Sept. 5th
Project Planning
Lab1 [due Sept. 19th]
3
Sept. 7th
Git Concepts 1
4
Sept. 12th
Git Concepts 2
5
Sept. 14th
Git Concepts 3
6
Sept. 19th
Git Concepts 4 + Presentation Session 1 (2 TIME SLOTS)
# Lab1 - Planning
Expect the best, plan for the worst, and prepare to be surprised! -Denis Waitley
Are you ready to embark upon a wonderful learning journey?
{% hint style="info" %}
*Here you can find the description of Lab 1 which is about project planning using Microsoft Azure. Enjoy the learning journey and don't forget to explore beyond the rules and regulations that are laid out in this article!*
{% endhint %}
## Creating an AZURE DevOps Organization
In this section, we cover the steps required to manually create an Azure DevOps organization
First, Navigate to the Azure DevOps website (dev.azure.com). Then, log in with your Microsoft account credentials and select New Organization in the left sidebar.
In the pop-up window, name your organization and set the region where you'd like your projects to be hosted. The name of the organization is ECSE437, and the region is set to Central Canada
On successful creation of the organization, Azure DevOps routes you to a page where you can create projects in your new organization.
## Creating an Azure DevOps Project
After creating the Azure DevOps organization, the Create A Project To Get Started page appears on the screen. To create a project using this form, specify a project name and visibility and then click Create Project.
After creating a project, you will see a dashboard with all the possible actions you can perform in the project. Azure DevOps projects are the starting point for everything you do.
Here is the Azure DevOps project dashboard
{% hint style="warning" %}
*NOTE: There is another way to create a new project. You can do this separately from creating an organization as well. There are many situations where you want to add a new project to an existing organization. Click your organization's name in the menu on the left. You will be redirected to the Organization Overview page. Click the New Project button in the upper-right corner.*
{% endhint %}
## Invite people to your team
If you click the blue invite button on the dashboard's top left corner, you can invite people to your team. Invite all the people on your team as well as me (Majid Babaei) and the TA (Sarvin: ).
## Create a work Item
Now, enter the details for the task. The following are the fields and their meanings so that it is easy for you to know what to type: (*Note: At this point, you don’t need to set all of these fields but later when you add the source code to your project you can add more detailed work item*)
* Title: Here, you can write the name of the task. (For example, create add-to-bag functionality for guest users.)
* Assigned: Here, you can assign work items to specific people on your team. This person must be a member of the project.
* Add tag: You can also add tags to this work item. You can use these tags to group related work items and find them later.
* State: Because this work item is new, the state is automatically set to To Do.
* Iteration: Here, you can define which sprint to add to this task. Setting the iteration can also happen later in the backlog.
* Description: Here, you can add more details about the task so that other people on your team have full context on the details of the work item.
* Discussion: Here, team members can post additional comments about the work item. You can have conversations here about tasks, pull requests, bugs, or any other thing relating to the work item.
* Priority: Here, you can set a number to indicate the priority of the work item.
* Activity: You can also classify this item. For this project, your work item can be classified as a deployment, design, development, documentation, requirements, or testing activity.
* Development: Here, you can link the item to a specific branch, build, commit, or Pull Request.
* Related Work: You can link the item to other items like GitHub issues and other work items. You can also create relationships between this task and others. Types of relationships include child, duplicate, duplicate of, parent, predecessor, related, successor, tested by, and tests.
Let’s create an issue and then some tasks based on this issue. The issue is called “learning the programming platform”. Then we can list all the items that we need to learn as tasks under this issue. Assign this issue to everybody on your team!
Now create a simple work item task called “Learning ReactJS” and assign it to all team members. Here is a sample description of the task and some tags, assign it to an iteration. (*You can give it a different priority point*)
If you go to the history option, you will see:
Now you can link this task to the issue we have already created. Click on the add link and the find the issue in the list. (Make sure the like type is *parent*)
Now you can see the link under the related work.
Now you can create a discussion on how to complete this task with your friends.
{% hint style="danger" %}
*Please do not mention me here!*
{% endhint %}
When you save this, you will see your work item like this:
{% hint style="warning" %}
*Note: You have successfully created a new work item. I recommend creating a variety of work items such as issues, epics, and tasks. This will help you become familiar with the different work items and how to navigate them.*
{% endhint %}
## Managing Backlogs
Backlogs help you plan your project with issues or epics. Once you have a plan, you can start working on writing the code for the product.
The product backlog is a roadmap of what your team plans to deliver. Backlogs have features that make project planning hassle-free. The following are some of these features:
* Quickly define tasks assigned to your team by identifying epics, product backlog entries, or issues
* Reorder your backlog to make sure you're working on the most important work items first
* Add details and estimates to your backlog items
* Quickly assign backlog items to team members and sprints
* Predict tasks to evaluate what can be delivered within a sprint
* The project dashboard gives you access to your backlogs. To visit the Backlogs page, select Boards ➪ Backlogs
Here, you can see all the other work items in the project. You can move work items into iterations (or sprints) from the backlog. (*It only shows you issues*)
In addition to moving backlog work items to iterations (or sprints), you can edit, reassign, copy, add relationships, or email work items. You can move work items between projects. This is especially useful when the organization is restructured, and projects are reassigned to various teams.
If you want to see the related work for this issue you need to click on it and check the list of the related work. (*You can add as much related work as you want here*)
If you click to view as a board, you will see the states that your issue can have (from to do to done).
When you start working on this issue you can drag and drop it to the “doing” state.
You can add several subtasks to this.
When you complete all the tasks you can move the issue to the done state.
When you open the backlog, you will see something like this:
Here you can check the status of the issue and each task.
## Sprints
Sprints (or iterations) are used to divide the work into specific periods. Teams use two or three weeks for their sprints depending on what works for them. This is based on the momentum that a team can endure, that is the rate at which the team is finishing the tasks.
Let's look at the Sprints page view in Azure DevOps in more detail:
* Select Sprints from the left menu below the dashboard. By default, the Backlog view is displayed. You can get an overview of the issues again here, except this time for the current sprint.
* Click Taskboard in the top menu to see different views of the work items in the sprint similar to what's happening in the board. This time, the items in the current sprint are shown at the task level in the backlog.
* From here, you can create new tasks under specific issues or in the unparented column; you can also drag, edit, and interact with work items as described in the “Boards” section.
Sprint task boards are often used by teams in their daily standups. Items are moved to different columns based on the team's progress. The team will also briefly discuss these elements and, if necessary, ask for help when there's a blocker or an error. At the end of the sprint, most items are moved to the Done column.
Let’s create a query:
You can filter work items based on filter criteria specified by Azure DevOps. This makes it easy to get an overview of all work items of a certain type, status, or tag. This can be done both within a project and across multiple projects.
To create various queries for searching through work items, complete the following steps:
* Click Query in the left menu under Boards.
* Next, let's create a query that will be searching for an issue with the Doing state.
* Then, click Run query. The result will display all the issues that are currently active
# Lab2 - Git
Don't expect your life to be as perfect as the main branch of your git project!
Doesn't feel marvelous to keep track of everything? getting back to history and only cherry-picking the best pieces of your past work and integrating them into the future?
{% hint style="info" %}
Git is a distributed version control system that tracks changes in any set of computer files, usually used for coordinating work among programmers who are collaboratively developing source code during software development. Its goals include speed, data integrity, and support for distributed, non-linear workflows. [Wikipedia](https://en.wikipedia.org/wiki/Git)
{% endhint %}
## Clone the repository in your local machine
First, we need to clone the repository and do some initial steps:
* Create a directory called “workspace” on your desktop and go to this directory
* Right-click and *open in* *Terminal*.
* Clone the repo using this command: git clone
* In the terminal go to the directory portfolio:
{% code fullWidth="false" %}
```
cd .\portfolio\
```
{% endcode %}
* Check all the files here using:
```
ls
```
* Check if you have a remote repository in git using:
```
git remote -v
```
* Yes, we have one called “*origin*”, which means we cannot add more with this name. If we want to add a new remote with the same name we need to remove this one.
* Let’s remove it!
```
git remote remove origin
```
* Now everything is ready!
#### Push your code into your DevOps project
Login to your Azure DevOps account: [*https://dev.azure.com/*](https://dev.azure.com/)
Then go to the *Repos* where you can import your files from your local machine, or you can directory clone from a remote repository.
You need to use the code generated for you in this part (*Note: the code shown in this picture may not be identical to what you see in your system because simply we run different projects!*)
What you need to do here is to run these commands in your terminal where your code is located!
When you run that you will see such message that tells you everything is pushed to your remote repository successfully!
That is good news! You can confirm that if back to your account!
Now you can get your code from your team repository in Azure DevOps Repos. If you click on the “Clone” icon, this window will pop up where you can get the address to clone the repo.
Everyone in your team can run this can clone the repo in their local machine:
* Copy the text in the box
* Open the terminal in the directory you want to place your code
* Run the *git clone* command with the address here
{% hint style="warning" %}
*Note: “ is the address on my project you need to use the address in your repo*
{% endhint %}
Open your code in VScode (you can download it from here: ) and run the code locally with the embedded terminal in vscode (ctrl + \~) and execute the *npm install* command:
```
npm install --save react-tinder-card --legacy-peer-deps
```
## Check and update the gitigonre file
*Note: This can be done only by one person in your team (all the people don’t need to repeat the process, because they can simply pull the changes!*)
Open the *.gitignore* file and check if all unnecessary files are already included in this file. It is important because when you run the code it will generate a lot of files and directories if you didn’t add these files, the generated files make your code messy. That reduces the maintainability of your code!
There question here is do we need to add “package-lock.json” to the .gitignore file or not?!
| According to the discussion in here: |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
Think about it, if you delete package-lock and re-install, you are forcing the latest versions of all packages in the dependency tree. Meaning, you are changing the behavior of (potentially) the entire application.
What are you really trying to do? If you have some weird npm-related problem, simply remove node\_modules and run npm install again. Removing package-lock.json is never the solution.
|
OK! Since we would want to force everyone to get the latest version of all packages, let’s add the “package-lock.json” file to this file and then update it!
Before committing the changes let’s get the *git status* and make sure is this the change we really want to do or we should take some extra steps.
The list of the files is OK but we don’t what to do this in the main branch! (*Why*?!)
Let’s create a new branch!
From the lectures, we know to create a new branch for our recent changes we need to pick up a name. So the next question is what is going to be the meaningful name for the branch?! At the end of the day (or sprint) we should be able to discuss all the jobs we have done on the projects. So it is better to provide such a link between the worktime we work on and the changes we have made in the code.
Let’s create a proper work item that reflects our intention.
{% hint style="warning" %}
*Note: It is nice to have a short description of the expected behavior of the work item (issue) especially if it is meant to fix a bug!*
{% endhint %}
* Don’t forget to assign it to yourself or someone who is in charge of performing the update
* Adding proper tags always is a good practice
* You can change the status to Doing once you start working on it, then other people will know what you are currently working on.
Now you can create a good name for your branch based on the issue you just created! The name could be a combination of the \[short format of] project name “PRTFL” and the issue number “10” and the issue name “Update gitignore file”:
```
PRTFL10/update-gitignore-file
```
{% hint style="warning" %}
*Note: You might get a different number for the issue*
{% endhint %}
Back to the code!
Run this to create the new branch:
```
git checkout -b PRTFL10/update-gitignore-file
```
Let’s check if everything is OK:
Yes! We are on the right branch and the modified file is already there.
All you need to do is:
* Add the file to your staging area
* Committing the changes to your local git repository
* Push the commit to the remote git repository
```
git add .gitignore
git commit -m "package-loc.json added to this file"
git push --set-upstream origin PRTFL10/update-gitignore-file
```
Nice! You have committed your first changes and let’s inform everyone that you have completed this task and celebrate!
But the question is: how can I inform everyone professionally about this small change? Do I need to reach out to them by email or phone or is there any systematic way to do that?
The short answer is Yes there is a better way to get your message across to your team!
Besides, you need to inform two groups of people:
* People who have access to the code and can verify your changes (Dev people)
* People who don’t have access to the code yet they eager to check and show off your achievement (Op people)
Create a Pull Request (PR) is a way to inform Dev people to let them know you are ready to get feedback on your changes.
Before creating a PR make sure you have done everything for this issue, if you want you can do more commits on the same issue before creating a PR.
{% hint style="warning" %}
*Note: Keep in mind that the size of your commit shouldn’t be very long, which means you should not modify multiple files in one commit because it makes it difficult for other people to understand the logic behind every commit and verify your changes*. *Do you wish to learn more? Checkout* [*this link*](https://softwareengineering.stackexchange.com/questions/206979/how-big-should-a-single-commit-be)
{% endhint %}
For this change we are OK, so we should create a PR.
* If you want you are welcome to add a more detailed description
* Pick a reviewer from your team who is acting such a role or capable of giving you constructive feedback (YOU CAN NOT BE A REVIEWER OF YOUR OWN CODE)
* Add a related working item if it is linked to the previous issues you fixed. It helps the reviewer to go through the chain of changes and get a better big picture of your attitude.
* If you have made your changes based on some suggestions or scientific papers you can add a link or those papers here as attachments. Again, it helps other people to understand your work better (*of course there are chances to show off your theoretical mind here*).
## Check the PR and merge to the main branch
Now other people in your team (especially the reviewer) can go to the pull requests and see the changes you have made.
In the files you can see the list of the changes in this PR:
You can add your comment on every line of each file and start the communication with the developer.
You can approve/approve with suggestions/reject/ect. the PR.
If there are no merge conflicts and after getting all the approvals from reviewers now its time to merge the changes to the main branch.
Click the *complete* and choose the best merging option. If you have made several commits to complete this issue (especially to address the feedback you have received), you’d better off squash the commits and don’t bring all of them to the history of the main branch! (*We will discuss the consequence of that during the lectures on VCS*)
For now, let’s stay with this option and do the merging!
If you are unhappy with the changes you can Revert it!
As you can see the associated work Items have changed to complete and this is how you can inform Ops people! (*But in this case, we didn’t add a link to this development work item so we should do it manually*)
## Update the other files using the same approach
How can I get the list of the files that need to be updated with our team information:
Select the project directory “portfolio” and then click *ctrl* and *h* at the same time and search for *\[…]* on the result you will see all the places you should update based on the information of your team.
For every file you should follow these steps:
* Create a work item
* Create a new branch
* Do the changes
* Commit to the remote repository
* Create a PR
* Link the work item to the PR
* Address the feedback from reviewers
* Merge the changes to the main branch
Make sure you include the description of the steps you have taken to complete this part in your learning journal.
# Lab3 - LCL
The Linux philosophy is 'Laugh in the face of danger'. Oops. Wrong One. 'Do it yourself'. Yes, that's it. --Linus Torvalds
Open Source-based technologies reduce the prospect of a product suddenly becoming obsolete. It allows for more collaboration and better innovation between industries.
{% hint style="info" %}
Linux is one of the most popular operating systems among professionals. In this article, you'll learn the most-used Linux commands. Also, you will be introduced to Docker!
{% endhint %}
## Introduction
In this Lab, you will learn how to work with Docker. Before talking about Docker, let’s take a moment to highlight containers. A container packages code and all its dependencies into a single unit, thus letting an application run quickly and reliably from one computing environment to another. This makes such applications easily portable between machines and solves the “it works on my machine” problem. Though the technology behind containers has been around for a while, Docker made it easier to work with containers. Since its debut in 2013, Docker has become an industry standard. Currently, the core technology exists as a popular, open-source container runtime called Docker Engine.
To create Docker containers, you’ll first need a Docker image. If you’re familiar with object-oriented programming concepts, think of images as classes and containers as objects. Images include everything needed to run an application: code, runtime, system tools, system libraries, and settings.
Docker is ideal for the following use cases, and many more:
* Software prototyping and packaging
* Microservice architecture implementation
* Network modeling
* Continuous integration and delivery
* Reducing debugging overhead
* Running more workloads on the same hardware
## Docker Desktop App
One of the best ways to get started with Docker is by installing Docker Desktop—especially if you’re a developer using Mac or Windows. That said, you might be wondering, “What’s Docker Desktop, and how’s it different from the open-source Docker Engine?”
While some developers envision Docker Desktop as just a GUI on top of Docker Engine, that characterization barely scratches the surface. Docker Desktop is an easy-to-install application and includes Docker Engine, Docker CLI client, Docker Compose, Docker Content Trust, Kubernetes, and Credential Helper. Docker Desktop still uses Docker Engine at its core. However, the seamless integration and interoperability of these tools make Docker Desktop user-friendly—regardless of your experience with Docker.
By installing and using Docker Desktop, you’ll enjoy the following features:
* Simple and easy-to-install environment to build, ship, and run your containers
* Easy way to create and manage using volumes
* Local and remote management of Docker images
* Better collaboration by sharing repeatable and reproducible development from your local machine to the container
* Simple, one-click Kubernetes setup for your local machine
* A dashboard for a quick overview of running containers, images, and volumes
* Support for building and using multi-architecture images
In the next section, we will see how to run Linux using the Docker Desktop App
## Running Linux
First head over to and complete the registration process, then login to your account
Then install the Docker desktop app from and log in with the credential you just created.
On the search bar, type Ubuntu and then Run its latest version:
When you Run the container, you will see this message in the terminal:
That means you need to run your container in interactive mode to be able to work with it. So let’s run the command prompt and then execute the following commands:
* Run the powershell
* Use: `docker ps` to see the running processes or containers
* Use: `docker ps -a` to see the stopped containers as well
* Finally, use: `docker run -it ubuntu` to run the container in an interactive mode
What we have here is called The Shell, It is a program that takes our commands and passes them to the OS for the execution. In this line, `root@982250960ecd:/#`, the root is the current user, after `@` we have the name of the machine, i.e., `982250960ecd`, automatically generated by Docker, after ‘:’ you can see where we are in the file system, i.e., `/`, which represents the root directory, finally we have `#` which means we have the highest privileges. If I logged in as a normal user instead of # you would see a `$` sign.
Next, we will explore some Linux commands.
## Working with Linux Command Line
Let’s see the location of the shell program. You need to run `echo $0` as you can see it refers to `/bin/bash` , bash is the enhanced version of the original shell program. Also using the history command you can see all the commands you have executed lately.
{% hint style="warning" %}
*Note #1: “shell” is a broad term that refers to any program that provides a command-line interface, "Bash" is a specific type of shell that is widely used in Unix/Linux systems.*
{% endhint %}
{% hint style="warning" %}
*Note #2: if you have noticed in Windows, we use a forward slash but in Linux, we use a backslash to separate directories.*
{% endhint %}
{% hint style="warning" %}
*Note #3: Linux is a case-sensitive operating system! There is no command “Echo” it is “echo”!*
{% endhint %}
## Working with Grep
Now we are going to learn about "grep" command. Grep stands for *Global regular expression print*. As the name implies, Grep is used to search text files with regular expressions (shortly regex). It prints the lines matching the given pattern in a text file. If no file is given, grep will recursively search the given pattern in the files in current directory. Grep has two variants, namely *egrep* and *fgrep*. These variants are deprecated but are provided for backward compatibility. Instead of using "grep -E" and "grep -F", you can use "egrp" and "fgrep", respectively.
First, let’s create a file with some random worlds and show its content.
`cat file.txt`
| Here is the output: |
| ---------------------------------------------------------------------------------------------------------------------------------------- |
|
ostechnix
Ostechnix
o$technix
linux
linus
unix
technology
hello world
HELLO world
|
To begin the search, just type grep followed by what it is you're looking for and where you are looking from. For example, I am going to look for the string "nix" in file.txt. To do so, I run:
`grep nix file.txt`
What did you get as the output?! (Include results in your learning journal)
{% hint style="warning" %}
*Note#4: You can also use -n flag to show the line numbers in the output. This can be useful when you're working with a really long code.*
{% endhint %}
Please note that grep is case-sensitive. What you will get if you run these commands:
```
grep os file.txt
grep -i os file.txt
grep -i 'hello world' file.txt
```
(Include results in your learning journal)
We can also pipe an output of a command into grep. Have a look at the following example.
```
cat file.txt | grep os
```
(Include results in your learning journal)
Now see what we've got. The output of the file.txt is piped into grep and the words that contain the letters "os" in file.txt have been displayed.
We can also use some special characters or regular expressions in grep.
* ^ - search at the beginning of the line.
* $ - search at the end of the line.
* . - Search any character.
What you will get if you run the following commands:
```
grep tech file.txt
grep ^tech file.txt
grep x$ file.txt
grep .n file.txt
```
(Include results in your learning journal)
You should now have a basic understanding of grep usage.
## Managing Users in Linux
In Linux we have some commands to add, modify, and delete a user:
```
useradd, usermod, and userdel
```
If you run this `useradd --help` you will see a bunch of options, you can use with this command. Here we only need `-m` or `--create-home` and let’s create the user called *john*.
```
useradd -m john
```
where is this user?! And how can I verify it?!
The user’s info is stored in a configuration file in `etc` directory. Let’s check it:
{% hint style="warning" %}
*Note #5: the name passwd is a bit misleading! Only user’s info will be stored here not their password!*
{% endhint %}
```
john:x:1000:1000::/home/john:/bin/sh
```
In this line “x” means the password is stored somewhere else! “1000:1000” refers to the user’s and its group id. Then we have “/home/john”, the home directory of the user, finally, we have “/bin/sh” the shell program used when this user logs in to this system.
But what if we want to use bash instead of shell when this user logs in? we should modify this information using `usermod` the command. First, let’s see what options we have:
Here I am going to use -s to modify the shell program.
```
usermod -s /bin/bash john
```
Now we can verify it!
Where are the passwords?! There is another file called shadow where all password is stored in an encrypted format. This file is only accessible to the root user!
To learn more about how this file works take a look at [this link](https://www.techtarget.com/searchsecurity/definition/shadow-password-file#:~:text=A%20shadow%20password%20file%2C%20also,from%20breaking%20into%20the%20system.).
Now that we have created the user, we can login to the container with this user. To do this, while your container is still running in a new tab in the command prompt run the following commands:
First, let’s check if your container is still running and get its id with `docker ps`
In my case, the id is: `217ef9e08bba`. You may get a different ID!
Then use the command exec to execute the bash program in this container with the user *john*
Then use the following command to log in with the new user:
```
docker exec -it -u john 217ef9e08bba bash
```
Let’s see what happens if I want to access the shadow file using john user!
This verifies that I am not a root user!
{% hint style="warning" %}
*Note #6: we also have an alternative command to create a user “adduser” it is more interactive than “useradd” and also allows you to set the password as well as additional information when you create a new user.*
{% endhint %}
## Managing Groups
First, let’s answer this question why do we need groups? All users in the same group will have the same permission to the system.
Let’s create a group called developers:
We can get its information in a configuration file in etc dir.
Now we want to add *`john`* to this group. Again we need to use `usermod` command but this time with different options.
As you can see we have two options: `-g` and `-G` what are the differences?
`-g` is used to modify a user’s primary group but `-G` is used to modify a user’s supplementary group.
* *Primary Group*: Used to decide which files are created by which users. One user must belong to only one group.
* *Supplementary Group*: Specifies one or more groups for users to share files between them. One user can belong to up to 15 secondary groups.
You can check out [this link](https://www.hostingadvice.com/how-to/linux-add-user-to-group/) to understand.
Here we want to modify john’s supplementary group:
We use the `groups` command to confirm this.
## File Permission
First, let’s go to the home directory and create a file called `deploy.sh` Files with this extension are called shell scripts. In these files, you can put your script and run it in one go!
To get permission for this file we should get a long listing:
* If the first letter is – it is a file. *d* means it is a directory
* We have 9 letters divided into 3 groups, e.g,. we have rw-r--r-- for the deploy.sh file. The first group is the permission for the user who owns this file. The second group is for the group that owns this file. The third group is the permission for everyone else.
* In each group we have *read*, *write*, and *execute* permissions. In john dire we have rwx that means we have full permission. We have *x* because we want to go into this directory.
If we try to execute the deploy.sh file we will get a permission error!
Because the root user does not have the execute permission.
Let’s add and then remove execute permission for the user who own this file with the command `chmod`:
We can do the same for the group and also others:
With that even john can execute this file! Because others have the *x* permission
We can also combine these in one command:
Here we removed x, r, and w from the group and the others.
# Lab4 - Docker
I cannot be contained because I am the container! --Jim Carrey
In the previous Lab, you learned how to install Docker and also you have started working with Linux Comand Line within Docker which I believe feels awesome to have such flexibility and freedom!
{% hint style="info" %}
In software engineering, containerization is operating system-level virtualization or application-level virtualization over multiple network resources so that software applications can run in isolated user spaces called containers in any cloud or non-cloud environment, regardless of type or vendor. [Wikipedia](https://en.wikipedia.org/wiki/Containerization_\(computing\))
{% endhint %}
## Let’s *Dockerize* the react-app
Let's start by adding a dockerfile to the main directory of your project. But what needs to be added to this file?!
As you have seen in the lecture this file contains several parts that need to be defined carefully (*otherwise it won’t work!*). Instead of jumping into a super-advanced fancy dockerfile, we adopt an incremental approach and add more fancy items as needed!
First, we need to pull the official base image. Since we use the node module, we need to make sure the base image contains a node and make sure the node version in our local machine is identical to the node version on the image.
The node version in my machine is 18.14.2.
To find out what base image is compatible with this version login to your account:
Then search for “node” and inside the sort by box type the version of your node module plus the name of the (minimal) OS, in this case, it is going to be: `18.14-apline`
What we need from this page is the complete name (tag name) of this image to refer to it in our `dockerfile: 18.14-alpine3.17`
Okay add this line to the dockerfile and then run the following command in the command prompt:
```
docker build -t portfolio-app .
```
If everything goes well you should be able to see this image in our docker desktop app:
As you have seen in t the lecture you can interact with this image using the shell:
It is a good time to brush up your memory about [linux file system](https://www.linuxfoundation.org/blog/blog/classic-sysadmin-the-linux-filesystem-explained)! You can also verify the version of the node running on this machine.
Now that we have the base image, the next step is to copy the application files into the image. Before that let’s define the working directory, i.e., *app*.
If you have noticed this process takes a lot of time! Essentially because it copies all the files in `node_modules` directory into the image! We should do something about it!
If the process is still ongoing, just stop it (*ctrl + c*) we don’t want a large image!
Let’s add the .*dockerignore* file beside our *dockerfile* and add node\_modules to it.
It is better now! Everything is done in less than 3 seconds!
Verify the results again by running the image:
Next, we install the dependencies on the image every time it is started and expose the port we would like to use to communicate with the image.
It is easy! The only thing we need to add to the dockerfile is:
```
RUN npm install
EXPOSE 8000
```
If you do that you will run into an error like this:
To understand what is going wrong on your image when you run npm install you should comment out these two lines and rebuild your image and then go to that and manually run the command:
You will need more options here!
Let’s go back to the dockerfile and fix the issue. If you open the README file you will get the additional arguments that need to be added.
```
RUN npm install --save react-tinder-card --legacy-peer-deps
```
Yes! It takes more time now. We will get back to this issue later!
We can verify that the *node\_modules* directory is generated in the image:
Since we would like to start the app once the container starts running, we need to add this command to the dockerfile: `CMD [“npm”, “run”, “start”]`
From the lectures, we have seen that the container is listening to port 8000 and it needs to be mapped to any port in the host, so we can get access to the ports on the container from the host. For that, we need to run the following script:
```
docker run -d -p 80:8000 portfolio-app
```
But when you try to see the portfolio from the host you will see the page like this:
That means it is not accessible from the host (YET)
To debug the issue first we need to make sure the port on the container is listening. For that, we need to install the `nload` package on the container and then run `nload -m` that to show us the incoming and outgoing data from our network. (*Try this and see the result*, also *check out* [*this link*](https://alpine.pkgs.org/3.15/alpine-main-aarch64/nload-0.7.4-r3.apk.html) *and* [*this link*](https://linux-tips.us/monitor-bandwidth-with-nload/) *for more information*)
According to [this link](https://www.parinda.dev/blog/gatsbyjs-with-headless-cms-part-1-04122020/) (*and many other resources!*), it seems since we use Gatsby in this project we need to some adjustments in the docker file to be able to see the result:
* We should use *yarn* instead of *npm*
* We need to explicitly mention the IP address of the host (i.e., 0.0.0.0)
Apply the modification in the dockerfile and then run:
```
docker build -t portfolio-app .
```
Finally, run the container with port mapping and check the result in your browser:
```
docker run -d -p 80:8000 portfolio-app
```
Everything works fine!
## Build and push Docker images to Azure Container Registry
First go to this address:
Then use the plus sign to create a resource and search for Azure Container Registry
Pick a name for the registry name, in this case, I use the registry name: `teamportfolio`
Use the default values for the rest and create the container registry.
Then click on “go to resource” and then use “Access Keys” to log in to the registry:
To log into the registry, you can use the script below:
Then get the list of all images and create an alias for the image with the proper name as below:
## Deploy the App
To deploy our app, we need to create a Web App using the production dockerfile.
First go to this address:
Then use the plus sign to create a resource and search for: Web App
Use the option “Web App for Containers” and create one.
## Oops, it doesn’t work!
So far everything has worked smoothly but now just in the very last step, it seems the web app we just created won’t work! Now it is your turn to put what you have learned as a software engineer into practice and try to fix the issue.
If you log into your portal, you can check the log files:
Please explain your solution and how did you find it in the learning journal!
Here are some hits that might come in handy as you try to debug the issue:
* Check the `portfolio-app` on your local machine and make sure it works OK. If it doesn't work maybe it is a good point to start!
* Check dockerfile and make sure its configuration is to be used in the web application.
* Check the platform we are using, in this case, Gatsby, do we need to do special treatment to support this platform in the web app.
* Check our web app configuration again, did we miss anything?!
# Lab5 - Pipeline
Let's rewrite our development process using a pipeline!
So far we have learned how to plan a project and collaborate with others on a software development project. We also worked with version control systems such as `git` to keep track of the files. in the previous Lab we explored Docker and deployed our team portfolio project on a cloud provider.
{% hint style="info" %}
A pipeline is a set of automated processes that allow developers and DevOps professionals to reliably and efficiently compile, build, and deploy their code to their production compute platforms.
{% endhint %}
## Create your first CI Pipeline
Let's log in to and then head over to “Pipelines” where you can create your CI and CD pipelines.
Click on “Create Pipeline” and then use the classic editor to create a pipeline without YAML. Select your repository. In this case, it is Azure Repos Git and then the branch “main”.
In the next step, we can choose a template or create an Empty pipeline and then add our jobs to it. Here choose to start with an “empty job”.
Create Agent job 1 and let’s add two npm tasks and publish artifacts task.
We need to change `npm` install step according to the install command we use in the package.json. If you don’t remember it you can go back to the files in the repository and get the command from there.
For the second step let’s rename the task npm run build under command choose from the dropdown “custom” and for the “command and argument” enter “run build”.
Since we use the Gatsby framework in this project we need to copy files from the public directory after building the project.
{% hint style="warning" %}
NOTE: In normal react projects, files will be stored in the *build* directory, and you don’t need to add this step.
{% endhint %}
The final step is to publish the build artifacts: Click on the plus sign look for “publish build artifacts” and set the “path to publish” to *build*, so that our release pipeline we will have access to the generated build.
Head over to the Triggers tab and select Enable Continuous Integration. You can leave everything else at default.
At the top select Pipeline and change the agent specification to ubuntu-last
Finally, click Save & Queue and again hit Save & Queue.
On the Run pipeline modal, leave everything at default and hit Save and run.
After a few minutes, the pipeline will finish running successfully. You can check under the Pipelines tab to verify it.
## Test the CI Pipeline
The only thing you need to do is change something in the code and commit to your master branch. But if you remember this is not how we do the changes!
First, you need to create a work item. For example, add the dockerfile to the project (if you have not done that otherwise do some random change in the read me file). Then create a branch for that and commit your changes in that branch. Finally, you can merge it to your master branch after getting approval from the reviewer!
{% hint style="warning" %}
NOTE: Do not add *package-lock.json* to your project. If you added this accidentally, go to the repository and remove it manually otherwise you will get some errors at `npm` build step.
{% endhint %}
Before doing the `git push -u origin` You will need to do `git pull`
Now you can go back to the Pipeline and see what is happening! Yes the CI pipeline is triggered itself and going into the steps one by one!
If you click on the link you can see the generated artifacts.
# Overview
Here you can find the list of all topics:
# Version Control Systems
{% hint style="info" %}
Here you can find presentations on Version Control Systems given by students
{% endhint %}
The list of topics is as follows:
# Topic#1 \[Taken]
Topic 1: [The Promises and Perils of Mining Git](https://ieeexplore.ieee.org/abstract/document/5069475)
{% hint style="warning" %}
This topic is taken by **Soumaia Bouhouia**!
{% endhint %}
{% file src="/files/KHHTsj2pgJfNGiuoYyxD" %}
“The Promises and Perils of Mining GitThe Promises and Perils of Mining Git” is a paper written in 2009 by Christian Bird\*, Peter C. Rigby†, Earl T. Barr\*, David J. Hamilton∗, Daniel M. German† and Prem Devanbu\* from the University of California (\*) and the University of Victoria (†).
Git, developed in 2005, was becoming increasingly popular at the time. This meant that more and more open-source projects were created on decentralized systems while, prior to that, most projects were hosted on centralized systems. This is significant on many levels, in particular in the domain of research as they may learn many things from a repository and its history. Here are 5 examples:
* They can reconstruct the process by which a software has been created. From that, some conclusions could be reached as to what steps developers go through to attain a publishable product, and the process could be made more efficient, thus gaining time and money.
* They can also study evolution patterns, meaning how a project is evolving as time passes.
* They can predict bugs.
* They can create recommender systems.
* They can explore collaborative processes.
As git was being used with increased frequency as time passed, it became important to determine whether the repositories created using Git would be as good for gathering data and performing analyses and how this analysis would need to change. Indeed, as we will see later, CSCM and DSCM systems like SVN (a centralized system) and Git (a decentralized system) have quite substantial differences that could either impede analysis or further facilitate it.
## **What is a CSCM system?**
CSCM stands for Centralized Source Code Management system. In essence, developers connect to a central source or server to access the repository, and one of the main characteristics of this system is that changing a file leads to only the difference (delta) being stored. While CSCM systems have many benefits, especially in terms of security, the aforementioned properties complicate the retrieval of previous versions of the code if the code on the centralized server becomes corrupted \[3]. SVN (Apache Subversion) is one such system, and the authors of the paper compared it to Git, a DSCM system.
## What is a DSCM system?
DSCM stands for Decentralized Source Code Management system. It enables developers to work independently on local repository copies, work offline while retaining access to the complete project history as well as create and merge branches at a minimal cost, and it enables them to commit individual changed lines within a file \[2].
DSCM data provides valuable insights for research. However, it also poses conceptual and practical challenges that are going to be expanded upon further in this summary. The paper's authors conducted a comparative analysis between SVN and Git, where SVN represents CSCM systems, while Git represents DSCM systems. They established a list of 9 promises Git makes, and compared the value of these promises with the perils Git was known to have at the time.
## Perils and Promises
In the following section, we examine the challenges associated with using Git, as identified by the authors. Each challenge is accompanied by a discussion of Git’s features that are instrumental in addressing these issues. The objective is to provide an informed and balanced perspective on how the features of Git counteract its identified shortcomings as described by the authors.
### Perils
1. **Git's Nomenclature:**
* ***Description:*** Git’s terminology differs significantly from traditional Centralized Source Code Management Systems (CSCMs). This distinction can create confusion and increase the learning curve for researchers wanting to perform analyses on Git.
* ***Mitigation (Promise 1 & 2):*** Any developer can make their repository public, creating a richer learning environment. The detailed information on commits, branches, and merges stored in Directed Acyclic Graphs (DAGs) facilitates an intuitive understanding of Git’s unique nomenclature over time.
2. **Automatic Creation of Implicit Branches:**
* ***Description:*** Git’s mechanism of automatically creating implicit branches can lead to complexity and confusion.
* ***Mitigation (Promise 2):*** Git's detailed recording of project history, including branches and merges in DAGs, helps in visualizing and understanding the development of implicit branches, simplifying their management and navigation.
3. **Analysis Methods:**
* ***Description:*** The utilization of DAGs instead of a mainline necessitates different analysis methods, potentially complicating the tracking of changes and history.
* ***Mitigation (Promise 2 & 3):*** The detailed data in Git’s DAGs and private logs allow for the development of refined, Git-specific analysis methods, offering deeper insights than traditional mainline analyses.
4. **History Rewriting:**
* ***Description:*** The ability to rewrite Git history can potentially lead to data integrity issues.
* ***Mitigation (Promise 4):*** The “signed-off-by” and other attributes ensure a detailed “paper trail,” enhancing traceability and accountability, thus minimizing the risks associated with history rewriting.
5. **Branch Commit Identification:**
* ***Description:*** Determining the specific branch on which a commit was made can be challenging.
* ***Mitigation (Promise 3 & 5):*** Private logs and explicit recording of contributor information enhance traceability, making it easier to identify the origins of specific commits.
6. **Tracking Merges:**
* ***Description:*** It can be complicated to ascertain the occurrence and location of merges.
* ***Mitigation (Promise 3):*** The private logs in Git contain intricate details, assisting in tracking and identifying merges, ensuring clarity in the merging process.
7. **Selective Commit Data:**
* **Description:** Accessible data may only contain selected commits, leading to incomplete information.
* ***Mitigation (Promise 1 & 6):*** Having developers' repositories and metadata publicly accessible ensures a richer dataset, reducing problems caused by selective commit data.
### Further Insights on Promises
* ***Promise 7:*** Git meticulously tracks file content and the history of lines, even as they are moved or copied, ensuring data integrity and providing detailed insights into the development of the codebase.
* **Promise 8:** Git is faster and uses less space than traditional SCMs.
* ***Promise 9:*** Most SCMs can be easily converted to Git while keeping the history intact, facilitating the adoption of Git's robust features.
## Empirical Evaluations of Git’s Perils and Promises
On top of listing these perils and promises, the authors evaluated some of them through empirical analyses using real-world data.
#### Promise 5: Enhanced Author Tracking
The authors checked if Git's ability to track authors influenced the number of contributors in a project. They compared projects using SVN and Git. The data showed that Git led to a significant increase in contributors, thanks to its improved author tracking.
#### Peril 6: Tracking Merges
The completeness of merge information in Git was examined using 30 open-source projects. The findings showed that Git could identify almost all (97.9%) of the merge sources, indicating its effectiveness. However, the authors recommended more in-depth research for a thorough understanding.
#### Git's Impact on Workflow
The authors also investigated whether Git changes how developers work. They guessed that developers might make smaller, more frequent changes after switching to Git from a centralized system. Their analysis showed that while there was a statistically significant difference in commit size, it wasn't a major change.
#### Promise 4: Understanding Contributions
The authors studied Git log messages to understand individual roles and contributions in projects better. In the Linux kernel project, for example, a specific developer was found to be signing off on most commits in certain areas, highlighting their key role and expertise. This detailed data is helpful in understanding the distribution of work and expertise in a project.
## Research Inquiries
In concluding their paper, the authors present further questions to delve into the extensive effects of moving to decentralized SCM systems like Git and its influence on team interaction and project results.
* Does changing from a centralized to a distributed SCM system affect how the project team communicates or develops the project?
* Does adopting a DSCM system encourage more focused development while possibly diminishing awareness of the broader project?
* Do developer teams sometimes work together separately from the main repository for extended periods?
A 2014 paper titled “How Do Centralized and Distributed Version Control Systems Impact Software Changes?” touches upon the first question. It provides insights into the impact of DSCM systems like Git on software modification patterns. The study reveals that Git promotes more frequent, yet smaller commits, attributed to the ease of making detailed change selections and the absence of conflict concerns with local repositories. Interestingly, projects that moved from SVN to Git didn’t show a change in commit size or frequency, suggesting that established CSCM commit practices were retained.
Additional observations from this 2014 paper include:
* ***Observation 1:*** Developers accustomed to CSCM systems expressed a preference for this workflow, attributing their comfort to familiarity rather than the system's features.
* ***Observation 2:*** There is a slight indication that commit sizes may reduce as team size expands, although this observation isn't backed by substantial evidence.
These insights offer a nuanced perspective on the adaptive behaviors and preferences of developers in the context of centralized and distributed SCM systems.
## References
1. Bird, Christian, et al. “The Promises and Perils of Mining Git .” IEEE Xplore, 5 June 2009, ieeexplore.ieee.org/abstract/document/5069475.
2. Brindescu, Caius, et al. “How Do Centralized and Distributed Version Control Systems Impact Software Changes?” ACM Conferences, 1 May 2014, dl.acm.org/doi/10.1145/2568225.2568322.
3. Zolkifli, Nazatul Nurlisa, et al. “Version Control System: A Review.” Procedia Computer Science, Elsevier, 29 Aug. 2018, [www.sciencedirect.com/science/article/pii/S1877050918314819](http://www.sciencedirect.com/science/article/pii/S1877050918314819).
# Topic#2 \[Taken]
Topic 2: [Can Git Repository Visualization Support Educators?](https://ieeexplore.ieee.org/abstract/document/9978497?casa_token=1zERXMOA7wEAAAAA:qcoRX5TCLq5jBYgrzLaTfnpPolVXjxuKR_0PNgxEPksrt600vOIW_sNKePfen_ERwvMA0lzB)
{% hint style="warning" %}
This topic is taken by **Felicia Sun**
{% endhint %}
{% file src="/files/w0T7s9lalvA2XFEC6YVJ" %}
This paper, published at the 2022 Working Conference on Software Visualization (VISSOFT), was authored by Mircea Lungu, Rolf-Helge Pfeiffer, Marco D’Ambros, Michele Lanza, and Jesper Findahl.
## 1. Introduction
The central premise of this paper is the exploration of how software visualization tools, primarily designed for software engineering professionals, can also prove to be invaluable assets in the realm of education. They aim to understand how these tools can aid educators in evaluating the quality of software systems developed by students. They also argue that educators are a special kind of user that can benefit from visualization tools.
Educators evaluating large projects in a short time have a different context than other stakeholders supported by visualization tools. Here are some examples of other stakeholders:
* Reverse engineers aim specifically at making sense of the code and usually have plenty of time to understand a system.
* Developers that are “onboarded” have ample time and also have access to the expertise of senior colleagues.
* Solo-developers who visualize their own software to identify improvement or refactoring candidates have deep expertise of their own code.
* Technology or domain experts have resources to learn and adopt specialized visualization tools over time.
## 2. Are educators a special kind of user for repository visualization tools?
Educators face some unique requirements and constraints:
* Educators need to assess large projects in a short amount of time. Those are often multi-person multi-month software systems that can easily reach tens of thousands of lines.
* The educator’s time is limited, and they can’t read complete solution. Also, the source code of the project is only one of multiple deliverables. They also need to look at the requirements, problem statement, user evaluation, experimental design, etc.
* Educators need to evaluate the individual contributions and not just the final result. Contributions can be unbalanced and they need to be able to easily spot any problems that might arise from this.
* An educator might also be teaching multiple courses with diverse technologies, or supervising multiple thesis projects simultaneously. This raises the need for a technology-independent tool.
* There are also have privacy concerns, since many educators might need to assess private or institutional repositories.
Following this, the authors of the paper have the following assumptions to carry out their research:
* Educators evaluate the students' code as part of the final assessment. They have limited time (e.g. 30 mins per group). In this time, they need to have at least a starting point for discussing with students.
* Projects are large and complex, making them impossible to understand completely in that limited time.
* Students work in groups and use file-based version control systems that track changes and their authors.
## 3. Using Git-Truck for Repository Visualization
The researchers used git-truck, a visualization which visualizes the structure of a git repository using hierarchical metric-enhanced layouts, such as circle packing visualizations or tree maps. The visual size of files is proportional to their size in bytes. Color maps are also used. The visualizations are highly interactive and support filtering, zooming, and presenting details on demand. Git truck supports author unification.
Git-truck is meant to be executed directly on personal computers from a local clone of a git repository ().
## 4. Usage Examples
Examples were selected by the researchers in collaboration with educators from:
* IT University of Copenhagen (ITU) in Denmark
* Università della Svizzera italiana (USI) in Switzerland
The usage examples are based on observations from 4 different courses by those educators. Note that the images mentioned can be found in the original paper.
### Usage Example 1: Finding components with a single author
Git-truck has a feature that highlights files that only had a single author in red. This allows educators to make fast assessments of the degree of actual group work going on, so they can quickly detect if there are components or even entire projects without collaboration amongst students.
In one image, the researchers looked at a part of the source code of a group project.
The visualization looked at is zoomed into the terraform folder in a larger group project. Note that Terraform is an infrastructure-as-code tool. The image suggests that although the group has five members almost all the work on this component has a single author. Later on, during the oral exam component of the course, they gave the other group members the chance to discuss concepts of infrastructure-as-code but only one could do so in any meaningful way.
### Usage Example 2: Investigating Responsibility Distribution in a Project
Another feature of the visualization tool can color-code the top contributor of each file, defined as the author who added or removed the most lines of code in the file throughout the history of the project. The goal is to support gauging how the work is distributed between members in the project or in specific areas of the project.
In the example repository, we have the Top Contributors view in the repository of another group from the same course as the other one. We can tell that all the members worked on all parts of a backend system, though the dark purple author is the top contributor for a lower amount of files.
### Usage Example 3: Investigating Responsibility Distribution in a Project
For another visualization, we can see a single author being the top contributor to most of the code files in the project. The goal of the project is that students get to experience the composition of a larger application from independently taught components. The educator in that course used this as a starting point for a discussion with the students about the individual contributions.
By discussing with the students, it was revealed that the "top contributor" is a group member who was new to Git and accidentally deleted the entire repository. Then they added everything back again. This serves as a valuable reminder to not blindly act based on the visualizations but rather discuss them with students.
Git-truck provides a rudimentary feature for selecting the last commit up to which the analysis can be done. With the help of this feature, an educator can spot projects where collaboration only becomes an issue after a certain commit.
### Usage Example 4: Gaining High-Level Architectural Insights
There’s also a feature to use color-coding to represent files based on their file extensions, while also visually scaling them proportionally according to their size and folder containment hierarchy. This view is useful to gain insights into the structure of systems with multiple languages involved.
The authors compared two projects in an intro web-dev course where the students had to implement the same front-end using React. The yellow files are for Javascript and the purple files are for CSS. The researchers observed some important differences between the way different teams would organize the files. Some groups made one big CSS file and many JavaScript files, while others decided to distribute CSS files across the system.
It was thanks to this visual representation that they were able to witness these architectural extremes and realize the importance of discussing file organization in future iterations of the course.
### Usage Example 5: Uncovering Critical Components of a System
With another visualization, the files are coloured according to a gradient. The files that are changed the most are the darkest. One of the authors of this paper browsed a repository before they had access to git-truck without spotting the god class that handled user interface interactions, database querying, scheduling, everything. Features like this one can be useful to spot bad design hidden very deeply into the directory tree.
## 5. Discussion
### Limitations:
There are some very clear limitations to these metrics-based visualizations. The researchers emphasize that educators should not rely on the visualizations without confirming hypotheses with students or triangulating with other sources.
There are instances where contribution metrics can be misleading, notably with:
* (1) automatically generated code and (2) files being committed that should not be tracked.
Besides this, there will always be unexpected usage patterns, such as the example where the student removes and adds all contents of a repository. So the takeaway here is that a repository visualization tool can highlight these unusual patterns but it is the duty of the educator to investigate them.
### Educator-Specific Tool Support
There are 6 key functionalities instrumental in increasing the adoption of such tools in education.
* Interactivity: Features like the ability to filter specific file types for visualization or zoom in on particular folders was very useful for assessing student projects. In the past, they relied on traditional git command line tools and various scripts, which are versatile, but not as user-friendly or as conveniently tailored for educational assessments.
* Author Unification: Students often use different usernames for commits due to variations in their git configurations across different devices, such as lab computers, private laptops, or home computers. Author unification can be done with git mailmap files, but students rarely configure them.
* Configurable Thresholds: The Single Author View has a 100% authorship threshold, but being able to customize this threshold, e.g. to 98% or other values, would be useful for a more tailored assessment option.
* Automatic File Filtering: Git-truck currently supports manual filtering of file types for visualization. However, this manual process can be time-consuming, especially when assessing multiple similar projects. Future research could explore automated detection of low-relevance files, using language-independent heuristics, such as identifying automatically generated code.
* Integration of Commit Messages: Currently, the educators have to review the commit messages alongside git-truck's visualizations. But ideally, commit messages should be integrated into the tool itself, making them easily accessible for educators.
* Support for Multi-Repositories: Educators frequently encounter both mono- and multi-repository projects, depending on course requirements. Many existing tools, including git-truck, primarily cater to mono-repositories. This limitation presents challenges when visualizing and comparing multi-repositories.
### Generalizability
Git-truck is designed with usability in mind. However, none of the visualizations it implements are unique or difficult to implement (though interactive features do require more effort). The usage examples presented earlier are likely to be useful for other educators using similar tools.
## 6. Related Work
Two separate researchers, Wattenberger and Tornhill, use circle packing, just like git-truck, to highlight metrics on top of repository structure. However, their work is not targeted at educators, but rather developers and business responsibles. Furthermore, although both provide online systems that could possibly visualize non-private GitHub repositories, none of the two services has the interactive nature argued for in this paper.
Two other researchers, Raclet and Silvestre, proposed Git4School, an analytics dashboard that enables a lecturer to follow the work of the students, commit by commit, to identify students experiencing difficulties. Their work is intended for a kind of educator and student in a context where the educator needs to guide individual student closely.
Kim et al. presented an interactive git repository visualization tool called Githru. They aim to support developers and domain experts in understanding a project's development history focusing on properties of the git commit graph. That is different from the goal presented in this paper.
Specialized tools focus on certain aspects of git repositories only.
* Cosentino et al.'s Gitana: Computes truck factors, identifies crucial authors for software projects.
* Gource (): Animates authors and their contributions over time.
* Git Timeline Generator (): Visualizes contribution frequencies over time.
* git-of-theseus: Creates static visualizations of repository growth over time.
* GitHub's built-in repository visualizations: Presents activity statistics, such as commit frequencies and number of contributors, among others.
## 7. Conclusions and Future Work
In conclusion, the authors of the paper argued that educators represent a distinct user category for software visualization tools. Multiple case studies from various courses and universities have demonstrated that git repository visualization can significantly aid educators in the assessment of group projects.
However, it's essential to acknowledge that the case studies presented are based on the experiences of a select group of educators who utilized a specific tool for group project assessment. The researchers plan to expand the research to include a wider group of educators. Their goal is to solidify the conclusions and also be able to offer clear guidance to educators and tool developers.
They are also hoping to delve into exploring the potential usage of such tools throughout the duration of a semester, since here they only researched using git repository visualization tools for the final evaluation phase. They hope this examination will provide a better understanding of the dynamic role of these tools in the educational process.
## 8. References
1. M. Lungu, R. -H. Pfeiffer, M. D’Ambros, M. Lanza and J. Findahl, "Can Git Repository Visualization Support Educators in Assessing Group Projects?," 2022 Working Conference on Software Visualization (VISSOFT), Limassol, Cyprus, 2022, pp. 187-191, doi: 10.1109/VISSOFT55257.2022.00030.
2. A. Tornhill, "Your code as a crime scene: use forensic techniques to arrest defects bottlenecks and bad design in your programs", Your Code as a Crime Scene, pp. 1-218, 2015.
3. A. Wattenberger, "Visualizing a codebase", 2021, \[online] Available: .
4. J.-B. Raclet and F. Silvestre, "Git4school: A dashboard for supporting teacher interventions in software engineering courses", European Conference on Technology Enhanced Learning, pp. 392-397, 2020.
5. Y. Kim, J. Kim, H. Jeon, Y.-H. Kim, H. Song, B. Kim, et al., "Githru: visual analytics for understanding software development history through git metadata analysis", IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, pp. 656-666, 2020.
6. V. Cosentino, J. L. C. Izquierdo and J.
# Code Review
{% hint style="info" %}
Here you can find presentations on Version Control Systems given by students
{% endhint %}
The list of topics is as follows:
# Topic#3 \[Taken]
Topic 3: [Modern code review: a case study at Google](https://dl.acm.org/doi/abs/10.1145/3183519.3183525)
{% hint style="warning" %}
This topic is taken by **Zach Hayden**
{% endhint %}
Full presentation: [ecse437\_topic3\_Zachary\_Hayden.pdf](https://github.com/zachary0249/ECSE437/files/12819414/ecse437_topic3_Zachary_Hayden.pdf)
Citation: Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. 2018. Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP '18). Association for Computing Machinery, New York, NY, USA, 181–190.
## Summary
1. Introduction
* The paper aims to address the current state of code review by taking a deep dive into Google's operations which have allowed them to scale to where they are today. Originally, code review was a very tedious and synchronous process wherein developers (including the author) would gather to discuss their code line by line with the goal of detecting erros in the code. This approach created more logistical and resource overhead. What the authors found to be the modern approach for code review, and a workflow that google had employed, was to use tooling to automate tasks and create organized ways of contributing asynchronously. This way, developers could review the code on their own time and not all together allowing for improved efficiency. Additionally, instead of reviewing code line by line they instead look only at the areas in which changes have been made. A common example of this that surrounds us in all types of development environments is the use of git pull requests which provide a means for teams to be able to review code seperately and communicate via comments to address issues.
* What motivated Google's code reviews weren't soley to identify errors in the program which had been the norm in the original code review construct, but instead the main focus was on the readability of the code base especially as it was rapidly growing. The reason for this the authors found was that the original Google developers wanted the code base to act as a strong education tool for the vast amounts of developers they would employ; by ensuring uniform code styling, making developers be language certified internally before being able to commit, and creating small commits, allowed Google to accomplish just that.
* Google made two key concepts vital for their code review: ownership and readability. Ownership in the sense that each directory is "owned" by a set of developers and this set were primarily in charge of writing and reviewing the changes made to their directory but only after these developers were trusted from a readability standpoint.
2. Approach
* The authors took a three-pronged approach for their research methods in order to extract what it is that defines Google's modern code review practices.
1. Semi-structured interviews with Google developers
2. Logs from Google's internal code review tool called Critique (only considering commits with more at least 1 additional reviewer)
3. Survey to Google employees who had recently made commits to gauge their perspective on the effectiveness of the process and how they feel it went for their most recent code change.
3. Results
* 80% of developers make < 7 changes a week
* Median developer makes 3 changes a week and reviews 4
* 80% of reviewers audit < 10 changes a week
* Median latency for entire code review is < 4 hours
* 35% of edits only modify one file
* Median number of lines modified in a change = 24
* < 25% of changes receive more than one reviewer
* 70% of edits are committed within a day of being published for initial review
* The authors identified key themes that were shown to cause friction in the process which were:
1. distance
> Distance is a challenge that many people are familiar with notably in the last couple years with covid. Being physically far from someone reviewing your code reduces streamlined communication pathways. Additionally inter-department reviews can pose as a distance of sorts where expectations and norms are different.
2. social interaction dynamics
> Social interactions of any type give rise to potential character conflicts such as using a demeaning tone or trying to exert their power / authority in a review exchange. This leads to overall frustration and reduced productivity.
3. subject matter
> Discrepancies in what each party thought the topic of discussion is can also lead to inefficiencies in the process. This is why it’s essential to clearly state the intention of the review especially for design reviews.
4. context of review
> There are varying levels of necessity for making a change and misunderstanding around the severity can cause communication problems.
5. cusomtization / configuration
> Differences in configurations and standards between ownership groups caused certain issues when cross-team collaboration was required. For example, when one team that required two reviewers had to collaborate with a team that required a number less or greator, there were disputes because of this miscommunication.
4. Takeaways and discussion
* Lighter is better
* Having only 1 reviewer improved productivity
* Much more efficient than original code inspections
* Less is more
* Having more frequent but less cumbersome commits is better for streamlining the entire process from edit to commit
* Large edits reduce quality of comments from reviewers and lengthens the code review process overall
* Reviewer recommendation systems are important in making sure that the code review process is effective as it is important to have someone who is familiar with the codebase to ensure that the changes proposed aren't harmful; these systems are an active area in current and ongoing research
* Static analysis integration is the most requested feature for code review tools according to data pulled by the authors
* Code review tools serve more purposes than just for code review alone, they can also provide meaningful data about the progress of the project during its development that can benefit developers as they work.
#### References
\[1] Rachel Potvin and Josh Levenburg. 2016. Why Google StoresBillions of Lines of Code in a Single Repository.Commun. ACM(2016). \[2] A.F. Ackerman, L.S. Buchwald, and F.H. Lewski. 1989. Softwareinspections: An effective verification process.IEEE Software6,3 (1989), 31–36. \[3] A.F. Ackerman, P.J. Fowler, and R.G. Ebenau. 1984. Softwareinspections and the industrial production of software. InSym-posium on Software validation: inspection-testing-verification-alternatives. \[4] M.E. Fagan. 1976. Design and code inspections to reduce errorsin program development.IBM Systems Journal15, 3 (1976),182–211.
# Topic#4 \[Taken]
Topic 4: [Who Should Review My Code?](https://ieeexplore.ieee.org/abstract/document/7081824?casa_token=6Z91gNG9zNIAAAAA:Z04ELyR7TMVN2rDI5q0HGnceQ7Y19xWl8eHdXorMDtfsP7hYd3PfN1G5MfEgf-Gem_4OUBFh)
{% hint style="warning" %}
This topic is taken by **Alexa Vasilakos**
{% endhint %}
## Overview of the Paper
“Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review” is a research paper by Patanamon Thongtanunam, Chakkrit Tanithamthavorn, Hajimu Iida and Ken-ichi Matsumoto from the Nara Institute of Science and Technology, Raula Gaikovina Kula from Osaka University, and Norihiro Yoshida from Nagoya University. The paper was published in the 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) which ran from March 2nd – 6th, 2015. The paper was added to IEEE Xplore on April 9th, 2015.
The authors aimed to address the following problem:
*“Finding appropriate code-reviewers is a necessary step in modern code review (MCR), however little research is known on the difficulty of finding appropriate code-reviewers in distributed software development and its impact on reviewing time.”*
The researchers wished to tackle a problem known as the “Code-Reviewer Assignment Problem.” They proposed a technology called REVFINDER, which is a file location-based code-reviewer recommendation approach that leverages previously reviewed file paths to determine and recommend appropriate code-reviewers.
## The Code-Reviewer Assignment Problem
The Code-Reviewer Assignment Problem is one of the main problems the researchers aimed to address. It refers to when an individual cannot find appropriate reviewers for their change request (we say that person has a “code-reviewer assignment problem”). In a Gerrit-based code-review system, a reviewer can either be identified as a “code-reviewer” or a “verifier”.
A **Code-reviewer** discusses the proposed change and suggests fixes.
A **Verifier** executes tests to ensure that the patch either fixes a defect or properly adds the feature that the owner of the change claims, and that the patch doesn’t cause regression of system behaviour.
Changes made by an owner will be integrated into the repository and marked as “Merged” if and only if it receives a code-review score of +2 (Approved) from a code-reviewer and it receives a verified score of +1 (Verified) from a verifier. In the Android example provided by the researchers, they depicted an owner (Smith) who was having trouble finding a code-reviewer for his change request, tasking his verifier (John) with finding one. Thus, it was concluded that Smith has a code-reviewer assignment problem.
## An Overview of REVFINDER
REVFINDER is a solution proposed by the researchers that leverages a similarity in previously reviewed file path to determine appropriate reviewers. The intuition is that “files located in similar paths would be managed and reviewed by similar experienced code-reviewers.”
The solution is composed of two parts: the **Code-Reviewers Ranking Algorithm** and the **Combination Technique**.
### 1. The Code-Reviewers Ranking Algorithm
This algorithm computes code-reviewer similarity scores via a similarity of previously reviewed file path. Given a new review as input, and a list of previously closed reviews, the algorithm will calculate a review similarity score for each of the previous reviews with the new review by comparing file paths using string comparison techniques. The scores are distributed to the code-reviewers involved, and the algorithm outputs a list of code-reviewers along with their scores.
The following four string comparison techniques were used. Underneath each technique is the assumption given by the researchers.
#### Longest Common Prefix (LCP)
**Assumption:** files under the same directory would have similar or related functionality.
#### Longest Common Suffix (LCS)
**Assumption:** files having the same name would have the same functionality.
#### Longest Common Substring (LCSubstr)
**Assumption:** since file path represents functionality, the related functionality should be under the same directory structure however, the root directories or filenames might not match.
#### Longest Common Subsequence (LCSubseq)
**Assumption:** files under similar directory structure would have similar or related functionality.
### 2. The Combination Technique
Due to variants in string comparison technique, REVFINDER combines the different lists into a unified list of code-reviewers with the idea being that truly relevant code-reviewers will “bubble up” to the top. **The Borda Count method** was used as the combination technique.
## Research Questions
The researchers aimed to address three research questions however, for the sake of time and relevance, I chose to explore research questions 2 and 3 in greater detail.
### Research Question #2:
*“Does REVFINDER accurately recommend code-reviewers?”*
### Research Question #3:
*“Does REVFINDER provide better ranking of recommended code-reviewers?”*
## Evaluation Methods
### 1. Studied Systems
REVFINDER was evaluated using four open-source software systems: Android (AOSP), OpenStack, Qt by Digia Plc., and LibreOffice. These systems were chosen by the researchers for two reasons: they use Gerrit for their code review system, and these are active real-world software systems that allow the authors to provide a realistic evaluation on REVFINDER.
### 2. Metrics
* Top-k accuracy
* Calculates the percentage of reviews that an approach can correctly recommend code-reviewers and the total number of reviews
* eg. a top-10 accuracy value of 75% indicates that for 75% of the reviews, at least one correct code-reviewer was returned in the top-10 results
* Mean Reciprocal Rank (MRR)
* Calculates an average of reciprocal ranks of correct code-reviewers in a recommendation list
### 3. REVIEWBOT – REVFINDER’s baseline
REVIEWBOT is another code-reviewer recommendation technology that operates off the assumption that “the most appropriate reviewers for a code review are those who previously modified or previously reviewed the sections of code which are included in the current review.” REVIEWBOT looks at line-by-line modification history to recommend code-reviewers.
## Results
### Research Question #2:
*“Does REVFINDER accurately recommend code-reviewers?”*
#### Approach:
For each studied system, REVFINDER was run on all reviews in chronological order to obtain the lists of code-reviewers, top-k accuracy was calculated, and results were compared against REVIEWBOT.
#### Result:
On average, for 79% of reviews, REVFINDER correctly recommended code-reviewers with a top-10 recommendation. REVFINDER ended up being 4 times more accurate than REVIEWBOT.
### Research Question #3:
*“Does REVFINDER provide better ranking of recommended code-reviewers?”*
#### Approach:
Mean Reciprocal Rank (MRR) was used to represent the overall ranking performance of REVFINDER, and results were compared against REVIEWBOT.
#### Result:
REVFINDER recommended the correct code-reviewers with a median rank of 4. The overall ranking of REVFINDER ended up being 3 times better than that of REVIEWBOT.
## Discussion
### Performance
*“Why does REVFINDER outperform REVIEWBOT”*
REVFINDER and REVIEWBOT are different when it comes to the granularity of code-review history, with REVFINDER leveraging file-path similarities and REVIEWBOT leveraging line-level similarities. The authors observed that 70% - 90% of lines of code are changed only once, concluding that line-level code review systems such as REVIEWBOT lack in performance.
### Applicability
*“Can REVFINDER effectively help developers find code-reviewers?”*
An exploratory study was performed where a representative sample of reviews was selected and then analyzed in order to identify which of them had code-reviewer assignment problem. The results of the study showed that reviews with code-reviewer assignment problem required more time to investigate the change. REVFINDER was executed on the reviews with code-reviewer assignment problem from the samples and found that on average, it correctly recommended code-reviewers for 80% of the reviews with a top-10 recommendation.
### Threats to Validity
#### Internal Validity
**Threat:** The reviews classification process was conducted by authors not involved in the code-review system.\
**Impact:** Since the classification process was conducted by authors not involved in the code-review system, this could be considered a threat, as they have decreased domain knowledge.
#### External Validity
**Threat:** The results are limited to four datasets (Android, OpenStack, Qt, LibreOffice).\
**Impact**: Limiting sample size in this way doesn’t give us the full picture, despite the fact that these systems were chosen due to their relevancy and high usage. Having a larger sample size provides a more realistic outcome on REVFINDER’s performance.
#### Construct Validity
**Threat:** Lack of code-reviewer retirement information.\
**Impact:** There’s is not enough information on code-reviewers who are retired or no longer involved in the code-review system. This can impact results greatly, as we might be recommended fantastic code-reviewers for the job, but they are no longer present. Thus, we have an inaccurate picture of which code-reviewers are truly available.
**Threat:** Code-reviewer workload.\
**Impact:** How can we measure how busy a code-reviewer is? It’s entirely possible that there’s a highly-ranked code-reviewer that often gets recommended by REVFINDER and is therefore burdened with many assigned reviews. Workload balancing would be a future consideration and optimization for REVFINDER.
## Future Work
The authors hope to deploy REVFINDER in a real development environment and perform experiments with developers to analyze the efficiency and practicality of REVFINDER in the workplace.
## Works Cited
P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida and K. -i. Matsumoto, "Who should review my code? A file location-based code-reviewer recommendation approach for Modern Code Review," 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Montreal, QC, Canada, 2015, pp. 141-150, doi: 10.1109/SANER.2015.7081824.
GitLab. “What Is a Code Review?” GitLab, about.gitlab.com/topics/version-control/what-is-code-review/.
# Containerization
{% hint style="info" %}
Here you can find presentations on Version Control Systems given by students
{% endhint %}
The list of topics is as follows:
# Topic#5 \[Taken]
Topic 5: [Developing docker and docker-compose specifications: A developers' survey](https://ieeexplore.ieee.org/abstract/document/9658534/)
{% hint style="warning" %}
This topic is taken by Sandy Nguyen, Minna Feng, and Béatrice Duval!
{% endhint %}
## 1. Introduction
Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure resources using code and automation instead of a physical hardware setup. In other words, it is about treating your computer infrastructure as something you can create, manage, and change using code.
IaC, especially with containers like Docker, has become popular due to cloud computing and the growing demand for efficient collaboration in DevOps. Docker is a widely used container technology that's become a standard in software development, leading the shift away from full-stack virtualization.
Because these technologies are becoming more important, we want to study their current usage, understand how people create and deploy Docker-based systems, and find better ways to address challenges. While there are many ancillary tools for infrastructure development, very few empirical studies show how they actually improve the development process.
**This presentation shows insights on what software professionals think about using Docker and helps find new ideas for solving current problems.**
## 2. Related Work
There are not many studies that look into the activities involved in creating container and orchestration specifications. Still, we are going to review relevant findings from existing research.
* A study by Guerriero et al. identified conflicting best and bad practices in IaC specifications, noting that available tools lack support for quality-focused maintenance and evolution. The main challenges also primarily revolve around testability and understandability.
* A study conducted by Haque et al. collected data from Stack Overflow and found that most of the Docker questions asked by developers fit into the categories of application development, configuration, networking, basic concepts and debugging.
* Cito et al. evaluated 560 open-source projects on GitHub and found that over a third of Dockerfiles couldn't be built error-free, and many had quality issues. This indicates that errors and quality concerns are frequent in Dockerfile development.
* A study by Rahman et al. showed that approximately half the IaC specifications in 2K IaC scripts contained some kind of syntax and configuration-related defect at one of their revisions, suggesting that this kind of defect may be more common in IaC than in non-IaC solutions.
* A study by Ibrahim concluded that more than a quarter of the projects analyzed needlessly used Docker-Compose, since they used one single component. They also found that the remaining multi-component specifications mostly use only the basic features of Docker-Compose.
* Some other works identify some opportunities for improving the efficiency of developing Docker specifications and for tools able to reduce the time spent in their development.
Many studies actually overlook technical aspects of container development, such as key activities, challenges, developer practices, and no studies focus on the most time-consuming tasks. Thus, significant limitations exist in maintaining and improving IaC specifications.
## 3. Goals of the Study
We hypothesize that the workflow of a developer configuring a Docker environment is often winding and based on trial-and-error, and that there is a need to study and improve these development activities.
This study answers three research questions in the context of working with Dockerfiles and docker-compose.yml files.
1. **Which activities are regarded as time-consuming?**\
By looking for these, we're searching for areas where developers can work more efficiently and identifying where we need better tools and methods.
2. **Which approaches are used to diagnose and correct problems?**\
Finding how people currently deal with problems can help us understand how to support them better or suggest new solutions.
3. **What role is played by ancillary software tools?**\
Ancillary tools are extra tools used alongside Docker and Docker-Compose. We hope that finding these tools will show us what developers need most when creating specifications for these platforms and where there might be gaps in the default support they offer.
## 4. Methodology
### Sampling
For the sampling methods, they used **convenience sampling**, a method that involves selecting individuals from the population that are accessible, and they also relied on **referral-chain sampling**, which is when one shares the questionnaire with their colleagues or surroundings.
To prevent oversampling specific subpopulation, the researchers decided to advertise questionnaire through varied experiences and contexts, that is: in **communities gathering** like Slack, Discord, Reddit; **Social networking** like LinkedIn, Twitter, Facebook; and also to **participants of the XP conference** which is the International Conference on Agile Software Development.
To persuade the participation in the survey, they offered the potential respondents **first-hand access to the results**, as well as showed a **short promotional video** that explained the goals of the study and the importance of participating.
### Questionnaire
In terms of the questionnaire, they made in available as an **online form**, where the response time was below 5 mins. They made sure to have precise and targeted questions including **mostly close-ended questions** and some open-ended ones to **identify issues and approaches** that might not have been anticipated in the close-ended questions. The first questions were to understand the participant’s background while the next questions were focused on either Dockerfiles or docker-compose.yml files. They only considered the respondents who reported having some experience creating or modifying those kind of files. As a result, they received **120 responses**.
## 5. Analysis of the Results
### Preliminary Run
Before sending the questionnaire to the sample, the researchers decided to have a preliminary run aiming to validate their questionnaire. They sent the questionnaire to 68 students and 2 researchers. The results show that the students perceive they spend much time understanding why a Docker container is not running as intended (54%) and that few participants (4.4%) use ancillary tools when working with Docker-Compose. Only 1.7% reported confusion of the questionnaire, indicating the researchers can proceed with submitting the questionnaire to the public.
### Demographics
The study gathered data from 120 participants, having diverse contexts and degrees of experience:
* Participants are from 24 different countries.
* Most work in the industry (90%), and some work in academia (12%).
* Most have responsibilities as software developers (87%) and many in operations (54%).
All participants have some years of experience with Docker technologies, working on projects that use these technologies, using these technologies directly (either using Dockerfile or docker-compose.yml files) or creating and updating such files.
The study considers two groups, **the experienced** and **the inexperienced** groups, on one hand having more than 3 years of experience, and on the other hand having less than 3.
### RQ1 — Which Activities are Time Consuming?
Researchers looked into both 1) dockerfile and 2) docker-compose.yml development processes and broke each down into several development activities. In both cases, researchers asked respondents about their attitude towards a considerable amount of time spent in each of these activities. They also looked into how responses may be affected by development experience.
#### Dockerfile Development Process
* A1 Reading Docker documentation.
* A2 Finding out what are the right Dockerfile Commands That I need.
* A3 Finding out what parent image is the most suitable.
* A4 Finding Out What Are The Dependencies My System That must be added to the docker image.
* A5 Confirming if the resulting container is working as intended.
* A6 Trying to understand why the resulting container is not working as intended.
* A7 Finding out which commands are responsible for the container misbehavior.
* A8 Rebuilding the image and re-running the container to confirm that it is working as intended
From this list of activities that were examined, it was found that developers tend to perceive as time-consuming most activities, especially **A4, A5, A6** and **A8**, the last 3 having to do with debugging. They also find that activities are perceived as less time consuming with experience, suggesting developers become more efficient as they progress through the learning curve of Dockerfile development. However, **A4, A5** and **A8** are regarded as time-consuming regardless of the years of experience.
Researchers suggest more focus should be put into improving activities **A4, A5, A6** and **A8** since they are perceived as the most time consuming globally.
#### Docker-Compose Development Process
* W1 Reading Docker documentation;
* W2 Finding out what are the keys that I need;
* W3 Finding out what images are available;
* W4 Trying to understand why the services are not working as intended;
* W5 (Re)starting the services to confirm that they are working as intended;
* W6 Configuring the properties of each service (e.g., port mapping, name, . . . );
* W7 Configuring the dependencies between the services(e.g., depends\_on);
* W8 Configuring volumes and how they are attached to the services;
* W9 Configuring networks and how they are connected to the services;
* W10 Configuring configs and how they are accessed by the services;
* W11 Configuring secrets and how they are accessed by the services.
Additionally, we have considered the following activities as concerning the reading of Docker-Compose specifications:
* R1 Trying to understand what the services are;
* R2 Trying to understand the dependencies between services (e.g., depends\_on);
* R3 Trying to understand what volumes are used and how they are attached to the services;
* R4 Trying to understand what networks are used and how they are connected to the services.
Results show that most activities seem to become easier with experience: activities are only perceived as time consuming by inexperienced developers. In particular, they consider the following activities as time consuming: configuring properties related to services (W6), and activities related to debugging (W4 and W5) and to documentation (W1 and W2).
**Researchers conclude the following:** Inexperienced users would benefit most from improvements in this area. This is consistent with the view that developers with less experience tend to use a trial-and-error method when debugging.
### RQ2 — Which Approaches are Used to Diagnose and Correct Problems?
Researchers asked what steps or strategies are being followed by respondents to diagnose and fix bugs in the creation of Dockerfiles and docker-compose.yml files. Surprisingly, both experienced and inexperienced developers have similar approaches
The approaches mentioned for problem solving in **Dockerfiles** include:
* trial-and-error
* searching Web resources
* entering the container to execute commands which may help to diagnose the issue manually
* continuous integration pipelines and
* end-to-end testing
For **Docker-Compose**, the results are similar:
* analysis of the output logs
* execution of commands within the running containers
* isolating services individually, making sure they work as expected
* testing their dependencies
What all these approaches have in common is that they imply leaving the environment where the specification is being written. The process could gain efficiency by introducing approaches that allow to gather feedback within the same environment.
### RQ3 — What Role is Played by Ancillary Software Tools?
In order to answer this research question, researchers asked participants if they use any plugins or tools other than a general-purpose IDE to develop Dockerfiles and docker-compose.yml files.
The results show that **tools are not widely used**, both by experienced and inexperienced users. Considering activities related to Dockerfile and Docker-Compose development are perceived as time consuming, the paper suggests there is a need for tools that improve the process of writing specifications, beyond what is possible with static analysis alone.
## 6. Threats and Validity and Limitations
There were many efforts to mitigate and identify threats to validity and limitations. They can be divided into 6 categories.
1. **Questionnaire design**\
Online questionnaires primarily used closed-ended questions, potentially limiting result explanations. However, the use of open-ended questions allowed researchers to capture more detailed responses.
2. **Clarity of the questionnaire**\
The questionnaire was designed for clarity and precision, but some participants may still interpret questions differently. Initial testing with students and researchers helped address this issue, with only 1.7% reporting confusion. The percentage is low enough to be dismissed.
3. **Honesty of the respondents**\
Researchers cannot guarantee that the participants answered the questionnaire in a truthful way. However, they believed that the participants had no incentive to give dishonest responses.
4. **Assessment of developer experience**\
Measuring experience based on years working with Dockerfiles may not be completely reliable since it depends on the participants’ memories as well as more time does not equal more experience. However, consistent and significant correlations in responses provided confidence in the precision of the measurement.
5. **Representativeness of the sample**\
It is impossible to guarantee that a convenience sample is representative of the entire population. However, the researchers expect the main conclusions of the work to reflect the current state of practice.
6. **Generalization to other IaC platforms**\
The study focused on Docker and Docker-Compose, limiting the scope of results. Despite this, the researchers believed their findings still hold relevance for a substantial number of professionals.
## 7. Conclusion
In conclusion, the study tries to reach insights on the development of container and orchestration specifications. The focus is on two IaC technologies, Docker and Docker-Compose.
The results were that the Dockerfile development process does present challenging activities and developers tend to perceive most of them as time consuming. The activities that would most benefit from improvements are finding the dependencies and debugging. In terms of Docker-compose, the results were not relevant although the activity that would most benefit from improvement is also debugging. Finally, most developers do not use ancillary tools in their development process.
## 8. Future Research
There are two main key takeaways from this research that can be made into potential future research.
* **Usability evaluation:** participants perform Dockerfile and docker-compose.yml development tasks and are observed to identify the major bottlenecks of the process, using methods such as task analysis or cognitive walkthrough.
* **New approaches or environments to develop Docker and Docker-Compose specifications:** to address the most time-consuming activities identified in this work and empirically demonstrate their merits and liabilities. Liveness, and live software development may play a relevant role in designing such environments, given their goal of proactively bringing feedback to users or even making the environment automatically act upon this feedback). We also consider that environments that enable visual programming and leverage model-driven engineering techniques can have an important role, especially for developers with less experience.
# Topic#6 \[Taken]
Topic 6: [Container orchestration: A survey](https://link.springer.com/chapter/10.1007/978-3-319-92378-9_14)
{% hint style="warning" %}
This topic is taken by **Arman Shroff-Mehrabadi!**
{% endhint %}
This presentation was based on the article, “Container Orchestration: A Survey” by Emiliano Casalicchio in 2019.
## Introduction
Modern cloud architectures are increasingly favoring container-centric designs over designs using virtual machines (VMs) across many different use cases. Like VMs, containers solve the “dependency Hell” problem (the issues with having numerous dependencies in large distributed applications), as well as being very portable. Additionally, they are generally faster, less resource-intensive, and easier to manage than VMs, which makes Containers a more popular choice going forward. As such, tools that manage containerized applications (container orchestrators) are more important than ever.
Container orchestration (using tools such as Kubernetes and Docker Swarm) is developing at breakneck speeds, but there remains many areas in which it can be improved upon. This article discusses the applications and designs of container orchestration technology, monitoring and performance evaluation of these tools, and research-proven features that should be added to orchestration tools.
## Types of Containers (System, Application, and Container Managers)
Containers are highly efficient because they share the host operating system's kernel, consuming minimal system resources and starting up quickly. This enables rapid scaling of applications, making them ideal for microservices architectures. System containers are a specialized type of container technology designed to encapsulate an entire operating system rather than individual applications. They are generally used for managing legacy applications. Virtually any new application would be built using application containers, which are the most common and what we have seen in class. They are lightweight, portable, and isolated units of software that encapsulate an application and all its dependencies, making it possible to run software consistently across different environments, from development to production. Container managers are frameworks that provide a set of APIs to easily manage the lifecycle (acquire -> build -> deliver -> deploy -> run -> maintain) of a container. Docker is used to manage application containers in a traditional server setup, while cloud providers also offer their own managers to do so on the cloud.
## Container Orchestration
Container orchestration tools such as Kubernetes or Docker Swarm further enhance scalability and fault tolerance, automating container management in large-scale deployments. Amazon, Microsoft, and other cloud providers also offer tools for container orchestration. Regardless of the flavor of technology used, they all serve the main purpose of enabling selecting, deploying, and dynamically controlling the configuration of multi-container applications. Container orchestrators handle numerous tasks, some of which are highlighted below.
* Resource limit control allows IT staff to reserve a specific amount of CPU and memory for a container. These constraints can then be used to make scheduling decisions and to limit the interference among containers.
* Scheduling defines the policy used to place the desired amount of container on desired nodes at a given time instant. Scheduling can be done on the basis of resource constraints or user configuration.
* The load balancer does the work of distributing the load among multiple container instances. Round-robin is the default implemented policy.
* Fault tolerance can be implemented as replica control and/or a high availability controller. Replica control allows to specify and maintain a desired number of containers. A health check, which is achieved by making containers’ TCP, UDP, or SSH ports open, is used to determine when a faulty container should be destroyed and a new one launched to maintain the target number of replicas.
* Autoscaling enables the automatic adding and removing of containers based on certain policies. This is one of the most important features of container orchestrators. The implemented policies are usually threshold based (by CPU and/or memory usage), but in some cases there are more sophisticated one or the option to define custom autoscaling policies.
### Current Limitations
There is an urgent need of complex adaptation policies to make orchestration a fully automated process with at least the following capabilities: Self-healing: to improve the application availability and resiliency Self-optimization: to balance the tradeoffs between cost and performance while fulfilling Service Level Agreements (SLAs) Self-protection: to increase system/application security
## Reference Architecture
Casalicchio describes an ideal architecture design, based off of the MAPE\_K (Monitor, analyze, plan, execute) self-adaptive cycle that IBM proposed for a self-adaptive auto-recovery system. This system monitors, diagnoses, checks and heals database applications automatically and immediately, without any human intervention. Casalicchio further describes his reference architecture as being multi-layered, in which the container platform layer (where the apps run) and the infrastructure layer (the hosting servers) are coordinated by the adaptation policy. This policy aims to find the optimal configuration, making decisions based on SLA requirements, increases in workload, customer objectives, node health, and other factors.
## State of the Art Research
There are three areas of container orchestration research Casalicchio highlights: monitoring & analysis and planning, the latter of which includes self-optimization and self-healing.
### Monitoring and Analysis
This section lists numerous studies that measure and analyze the performance of containers versus VMs and other platform technologies for different applications. Some of the studies are concerned with the methodology (and difficulties) in testing the performance of containerized applications. One somewhat thorough study (Felter et. al, 2015) performed a variety of workloads (some were CPU-intensive, network-intensive, or I/O-intensive) and found that by all benchmarks containers outperformed VMs. It should be noted that the difference is often less than 10% and VM technology has been improving relative to container performance. Another interesting study (Morabito, 2016) measured the overhead for using containers in IoT devices (which have very constrained processing abilities) and found that, on Raspberry Pi’s at least, the overhead was negligible.
### Planning
In general, automated container orchestration should be modernized to address the problem of auto-scaling, which is a key component of self-optimization. New automated management solutions should be developed. For instance, it can be difficult to deploy containerized applications across multiple cloud providers, which would be desirable due to rising costs (to the point where it can be cheaper to move away from the public cloud). A study (Abdelbaky et. al., 2015) developed a prototype framework, which would make deploying containers across several different cloud providers (e.g. Azure, AWS, and GCP) simultaneously relatively straightforward. It is not tied to any one provider, nor to a specific orchestration tool so would not only work with Kubernetes.
Other studies found ways to improve the performance of deployed applications. One study of particular interest (Al-Dhuraibi et. al., 2017) developed “ELASTICDOCKER”, which automatically vertically scales up and down allocated CPU and memory for each container. They found it increased the performance of the deployed application by more than 37% when compared to Kubernetes’ traditional horizontal scaling. Examples such as this reflect the importance of container orchestrators incorporating features proven from more recent research.
As for self-healing, the author briefly discusses two papers that develop prototypes to enhance the dependability of container orchestration tools. Using computational intelligence (i.e. artificial intelligence), it is possible to predict the possible failure of a managing node on Docker Swarm so that it can be replaced before causing the deployment to crash.
## Final Remarks
Many of the mechanisms for container orchestration could be improved for runtime self-adaption, with the three areas below being most prescient.
* Monitoring and Workload Characterization: monitoring techniques and tools used for the at the OS and application levels do not measure the performance behavior of containers. Moreover, there is no a commonly agreed definition of QoS metrics for container-based systems.
* Performance Models: Validated performance models and energy consumption models of container-based systems are inexistent. Performance models are widely used in autonomic computing to determine the reconfiguration actions needed to maintain the desired level of service.
* Adaptation Models for Container Orchestration: As already pointed out, the container orchestration policies used until now are very simple. A framework for QoS-aware, energy-aware, and legislation-aware optimal adaptation models is needed. This framework should allow the user to define system models, QoS, energy, and legal constraints, which it would then use to find optimal adaptation policies for container orchestration at runtime.
## References
M. Abdelbaky, J. Diaz-Montes, M. Parashar, M. Unuvar and M. Steinder, "Docker Containers across Multiple Clouds and Data Centers," 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC), Limassol, Cyprus, 2015, pp. 368-371, doi: 10.1109/UCC.2015.58.
Y. Al-Dhuraibi, F. Paraiso, N. Djarallah and P. Merle, "Autonomic Vertical Elasticity of Docker Containers with ELASTICDOCKER," 2017 IEEE 10th International Conference on Cloud Computing (CLOUD), Honololu, HI, USA, 2017, pp. 472-479, doi: 10.1109/CLOUD.2017.67.
Casalicchio, E. (2019). Container Orchestration: A Survey. In: Puliafito, A., Trivedi, K. (eds) Systems Modeling: Methodologies and Tools. EAI/Springer Innovations in Communication and Computing. Springer, Cham.
R. Morabito, "A performance evaluation of container technologies on Internet of Things devices," 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), San Francisco, CA, USA, 2016, pp. 999-1000, doi: 10.1109/INFCOMW\.2016.7562228.
W. Felter, A. Ferreira, R. Rajamony and J. Rubio, "An updated performance comparison of virtual machines and Linux containers," 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Philadelphia, PA, USA, 2015, pp. 171-172, doi: 10.1109/ISPASS.2015.7095802.
# Topic#7 \[Taken]
Topic 7: [Machine learning-based orchestration of containers: A taxonomy and future directions](https://dl.acm.org/doi/abs/10.1145/3510415)
{% hint style="warning" %}
This topic is taken by Hadi **Ghaddar and Omar Marwan!**
{% endhint %}
This presentation was inspired by the paper Machine Learning-based Orchestration of Containers: A Taxonomy and Future Directions, published on September 13 2022 and authored by Zhiheng Zhong, Minxian Xu, Maria Alejandra Rodriguez, Chengzhong Xu and Rajkumar Buyya.
## Introduction
In the past decade, cloud computing has gained popularity for its efficiency and scalability. Leading cloud services like AWS, Google Cloud, and Azure, initially relying on Virtual Machines (VMs), have transitioned to containers. Unlike VMs, containers don't need a separate operating system, making them more lightweight and efficient. This shift has prompted companies to favor containerization over VMs, leading to the rise of container orchestration techniques for managing cloud-based applications.
Container orchestration automates the lifecycle of containers, including the deployment, management, and scaling of containers without worrying about the underlying infrastructure. This proves advantageous for cloud providers handling numerous containers simultaneously, improving resource utilization and application performance. However, as the demand for cloud workloads grows, container orchestration techniques need to be enhanced to meet future expectations.
To meet these demands, companies are turning to Machine Learning (ML) techniques. ML algorithms help create complex models that understand and predict application metrics like resource utilization. This enables more accurate and efficient resource provisioning decisions, optimizing workloads in the ever-evolving cloud computing landscape.
## Background
Containers have a history that traces back to the 1970s, with roots in technologies like chroot Unix operation, which was the first glimpse into process isolation. However, their widespread adoption began in the 2010s with the introduction of Docker, and they have been rising in popularity ever since.
* Containers are lightweight packages of application code bundled together with dependencies such as specific versions of programming language runtimes and libraries required to run software services.
* Containerization provides a clear separation of responsibility, as developers focus on application logic and dependencies, while IT operations teams can focus on deployment and management instead of application details such as specific software versions and configurations.
* Containers can also run virtually anywhere, greatly easing development and deployment.
* Containers virtualize CPU, memory, storage, and network resources at the operating system level, providing developers with a view of the OS logically isolated from other applications.
As application architectures become more complex and the number of containers needed to maintain stability across a distributed system grows, companies, such as cloud service provides, can simplify the management of their container infrastructure with container orchestration.
* Container orchestration empowers cloud service providers by allowing them to capture the benefits of containerization at scale without incurring additional maintenance overhead by automating the complete lifecycle of containers, including configuration, deployment, and scaling
* Container orchestration simplifies operations through automated container management, can automatically restart or scale a container or group of containers, and add security by reducing the chance of human error.
An example of a leading orchestration tool is the widely used Kubernetes, which is an open-source platform developed by Google that helps companies build and manage containerized applications across multiple hosts, dynamically deploying, scheduling, and scaling them.
For future needs (and even some upcoming), current orchestration techniques are simply not equipped to deal with them, and enhancements to these techniques will be required to keep up with industry & customer expectations.
Machine learning is a method of data analysis that automates analytical model building. It is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns, and make decisions with minimal human intervention. Machine Learning presents the following benefits:
* It can handle and process large amounts of data.
* It has improved accuracy & efficiency over standard methods through training.
* It requires little to no human interaction.
* It can form accurate predictions on data through pattern recognition.
* It can be applied in all facets of software delivery.
Machine learning can be leveraged to perform complex calculations to predict trends, and we can use this in tandem with container orchestration ; through a machine learning-based Optimization Engine, we can build ML models based on analysis of monitoring data and system logs received from the orchestrator. Furthermore, it can produce future resource provisioning decisions relying on the generated behavior models and prediction results.
## Machine Learning Based Container Orchestration Techniques
Containerized applications can be composed through three different types of architecture:
| Architecture | Description |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Monolithic | Monolithic applications follow an all-in-one architecture where all the functional modules are developed and configured into exactly one deployment unit, namely, one container. However, it is hard and impractical to scale these applications to higher complexities as testing and development becomes resource intensive, thus making the architecture exclusively suitable for small-scale applications with simple internal structures. |
| Microservice | Microservice applications break down a complex application into multiple loosely coupled components called microservices. Each microservice unit can be deployed and operated independently for different functionalities and business objectives, and they can interact with each other. With each microservice unit, a new container must be implemented and maintained, which can cause system resources to plummet if not well managed. |
| Serverless | Serverless applications allow developers to build and run services without having to manage the underlying infrastructure. Developers can write and deploy code, while a cloud provider provisions servers to run their applications, databases, and storage systems at any scale. However, when a function hasn’t been used for a while, it is killed. This means that when it is called again, the time it takes the code to run is temporarily slowed, also known as a cold start, which could affect performance. |
Machine Learning can thus help us leverage efficient solutions depending on the architecture used by our workload. Within these workloads, ML can enhance various aspects of the application and infrastructure through multiple means:
* Time series analysis can be used to predict request arrival rates and resource usage patterns, allowing us to predict when and how many new containers should be provisioned or terminated, which can help optimize resource allocation. This analysis also allows our Machine Learning engine to dynamically allocate resources to containers based on real time demand and metrics. This ensures that resources are efficiently used and distributed effectively to allow applications to run smoothly.
* Anomaly detection is a critical mechanism for identifying abnormal behaviors in system states or application performance, and ML algorithms can utilize this by monitoring the behavior of containers, since cloud computing is subject to security attacks and threats. Through the identification of critical performance indicators that support anomaly detection, including network activity and CPU utilization, companies can identify security breaches instantaneously at any time.
* Load balancing between microservices under the multi-cloud environment could be rather complex, because of the unstable network latency, dynamic service configuration, and fluctuating application workloads. To address this challenge, ML can produce optimal request chains for each microservice unit to ensure proper queueing and resource management, dynamically updating the requests chains in case of environmental changes.
* Task scheduling is an issue in all OS based systems, and ML can help us alleviate these issues, as we can use approximation algorithms such as best fit to make scheduling decisions with improved resource utilization and energy efficiency.
* ML can predict when containers or infrastructure components are likely to fail or require maintenance. This can help orchestration platforms schedule proactive maintenance or failover procedures, minimizing downtime in cloud computing software solutions.
## Conclusion
Cloud Computing has revolutionized software delivery, offering cost-effective and scalable solutions to deploy and manage solutions. Cloud service providers have made the switch from Virtual Machines to Containers, which package applications and their dependencies, making them lightweight, efficient, and highly portable without maintaining an OS. In consequence, orchestration techniques have emerged as the preferred method for managing containerized applications, automating deployment, scaling, and maintainability. As the demand for bigger and better cloud workloads continues to surge, container orchestration must evolve to meet these standards, which can be possible with Machine Learning.
Through Machine Learning’s use of data and algorithms to make predictions and recognize patterns, we can integrate Machine Learning based optimization engines into orchestration to create models that can predict and optimize resource allocation and provisioning. ML-based models could produce more accurate orchestration decisions with shorter computation delays, under complex and dynamic cloud environments consisting of geographically distributed computation resources.
ML-based orchestration approaches and will also help to select the most suitable techniques for efficient container orchestration with the specific requirement under different application architectures, such as microservices and serverless architectures to handle the sustainably growing system complexity and balance multi-dimensional optimization objectives.
Through this synergy, companies will be able to create more scaled and robust cloud computing services which will help benefits users and businesses alike, opening a new frontier for new potential ways of accessing services through the web.
## References
Zhong, Z., Xu, M., Rodriguez, M. A., Xu, C., & Buyya, R. (2022, January 1). Machine learning-based orchestration of containers: A taxonomy and Future Directions. ACM Digital Library.
Google. What is container orchestration? Google, .
Docker. What is a Container? Docker,
IBM. What is Machine Learning? IBM,
# Pipeline
{% hint style="info" %}
Here you can find presentations on Version Control Systems given by students
{% endhint %}
The list of topics is as follows:
# Topic#8 \[Taken]
Topic 8: [The Impact of a Continuous Integration Service on the Delivery Time of Merged Pull Requests](https://arxiv.org/abs/2305.16365)
{% hint style="info" %}
This topic is taken by **Ningning Yang**!
{% endhint %}
Full presentation: [ecse437\_topic8\_Ningning\_Yang.pdf](https://github.com/Ningning-Yang/Ningning-Yang/blob/94cf11611e9f6edc318ff8902407142d1667309f/ECSE437_CI_in_Software_Development.pdf)
## Introduction & Background
Introduction
* Overview of CI:
* CI, as highlighted in the paper, is a software development practice involving the automatic integration of code changes into a shared repository.
* Emphasizes automated testing, aiming to detect and address issues early in the development process.
* Motivation:
* The paper emphasizes the motivation behind CI adoption as a means to quicken the delivery of merged Pull Requests in software projects.
* The focus is on understanding how CI impacts the software development lifecycle, specifically in terms of PR processing and delivery time.
* Significance:
* While the adoption of Travis CI, a CI service, doesn't universally accelerate the delivery of merged PRs, the study reveals crucial insights.
* CI, as a practice, significantly influences the decision-making processes of software projects, affecting code review and release procedures.
* Automation and increased confidence are identified as key benefits, impacting project throughput and contributor engagement.
Background of Study
* The study focused on 87 GitHub projects employing TRAVISCI as their chosen CI service.
* TRAVISCI is known for automating builds and testing processes upon code changes.
* Quantitative Analysis
* Investigated the impact of TRAVISCI on the delivery time of merged pull requests across these projects.
* Examined factors influencing the effectiveness of CI using a vast dataset of 162,653 pull requests.
* Qualitative Insights
* Complemented the quantitative aspect with a qualitative study, tapping into the perceptions of contributors involved in 73 of these projects.
* Explored how CI influences code reviews, release processes, and the attraction of external contributors.
## Quantitative Study
Questions:
* Q1: Are merged pull requests released more quickly using a CI service?
* Q2: Does the increased number of PR submissions after adopting a CI service increase the delivery time of pull requests?
* Q3: What factors impact the delivery time after adopting a CI service?
Findings:
* Q1: Only 51.3% of projects showed quicker delivery of merged PRs after adopting TRAVISCI.
* Q2: 54% of projects experienced longer PR lifetimes after CI adoption.
* Q3: Increased PR submissions and merge workload were significant factors affecting delivery time.
## Quanlitative Study
Questions:
* Q4: What is the perceived influence of CI on the time to deliver merged PRs?
* Q5: What are the perceived causes of delay in the delivery time of merged PRs?
* Q6: What is the perceived influence of CI on the software release process?
* Q7: What is the perceived influence of CI on the code review process?
* Q8: What is the perceived influence of CI on attracting more contributors to open-source projects?
Findings:
* Q4:
* Participants expressed mixed views regarding the impact of CI on the delivery time of merged PRs.
* While some noticed an improvement in delivery speed, others did not observe significant changes.
* This aligns with the quantitative finding that only about half of the projects experienced quicker delivery post CI adoption.
* Q5:
* Contributors often cited increased complexity and volume of PR submissions as contributing factors to delays in delivery time.
* This observation is consistent with the quantitative result showing an increase in PR lifetime in 54% of the projects after adopting CI.
* Q6:
* Many respondents felt that CI facilitated a smoother release process by enabling more consistent and automated testing.
* However, they also noted that this did not necessarily translate to an increased release frequency, as the study's quantitative results similarly indicated.
* Q7:
* Participants reported that CI helped streamline the code review process by automating certain checks, leading to more focused and efficient reviews.
* However, the faster merging of PRs before CI adoption in 73% of projects suggests that CI's role in speeding up the review process is complex.
* Q8:
* Contributors believed that the adoption of CI practices made the project more appealing to potential new contributors by demonstrating a commitment to modern development practices and ensuring code quality.
* This qualitative insight complements the quantitative finding that 75.9% of projects saw an increase in the number of contributors post CI adoption.
## Implications
Practical Implications for Software Teams
* Considered CI Adoption:
* Teams should adopt CI services like TravisCI with a clear understanding of their potential and limitations.
* While CI can improve aspects of the development process, it may not always speed up the delivery time of PRs. Teams should evaluate how CI aligns with their specific project needs and workflows.
* Focus Beyond Speed:
* The findings suggest that the benefits of CI extend beyond just delivery speed.
* CI can enhance decision-making processes, improve code quality, and facilitate handling a larger volume of contributions.
* Teams should leverage these advantages while being aware that delivery speed may not be significantly impacted.
* Adaptation to Increased Workloads:
* With the adoption of CI, teams might face an increased number of submissions, leading to longer PR lifetimes.
* Software teams should prepare for this potential increase in workload and consider strategies to manage it effectively.
* Quality over Quantity:
* The study suggests that a strategic focus on quality and efficient management of PRs could be more beneficial than simply aiming for speed.
* Teams should prioritize maintaining high-quality code and effective review processes over merely expediting PR delivery.
* Shorter Release Cycles:
* To optimize the delivery process, teams might consider adopting shorter release cycles.
* The study indicates that merged PRs have a lower delivery time when they are merged more recently in the release cycle.
## The Validity & Limitation of Study
Validity of the Study
* Comprehensive Data Analysis:
* Utilized a large dataset of 162,653 pull requests from 87 GitHub projects, ensuring a broad and diverse sample.
* Mixed-Methods Approach:
* Combined quantitative and qualitative research methods, providing a well-rounded perspective.
* Empirical Evidence:
* Offered empirical evidence to support findings, enhancing the study's credibility.
* Relevance to Current Practices:
* Addressed a highly relevant topic in modern software development, making the findings applicable and timely.
Limitations of the Study
* Focus on TravisCI Only:
* Limited to projects using TravisCI, not encompassing other CI tools which might yield different results.
* Restricted to GitHub Projects:
* Exclusively analyzed GitHub projects, potentially missing insights from projects hosted on other platforms or in private repositories.
* Potential Biases in Survey Responses:
* Reliance on self-reported data in surveys could introduce subjective biases.
* Open-Source Project Emphasis:
* Primarily addressed open-source projects, which may not completely represent the dynamics in closed-source or commercial software environments.
* Generalizability Concerns:
* The findings might not be entirely generalizable to all software development contexts due to the specific focus and sample used in the study.
## Conclusions
The study on the impact of CI services like TRAVISCI on the delivery time of merged PRs offers new insights that challenge prevailing assumptions in software development.
While CI brings several advantages, its impact on speeding up PR delivery is not as straightforward as often believed.
Teams must approach CI adoption with a nuanced understanding, focusing on its broader benefits and preparing for potential increases in workload. By doing so, they can effectively harness CI to enhance their software development processes.
## Reference
Bernardo, J. H., da Costa, D. A., Kulesza, U., & Treude, C. (2023). The impact of a continuous integration service on the delivery time of merged pull requests. Empirical Software Engineering, 28(4).
# Topic#9 \[Taken]
Topic 9: [Learning-to-rank vs ranking-to-learn: Strategies for regression testing in continuous integration](https://dl.acm.org/doi/abs/10.1145/3377811.3380369?casa_token=589-48O3V2YAAAAA:ZHmEK7dF7uSAaucGLiSxQJPDa_EmDpmSByQIRa_itN02J3YsacGJ26cHo6Ns2AEuQREYlw3A7Q57)
{% hint style="warning" %}
This topic is taken by **Biruk berhanu Retta**
{% endhint %}
## Introduction
The paper aims to compare different machine learning algorithms in respect to their effectiveness during selection and filtering of test suites. It also takes into account different experimental factors such as variability of code and time between commits.
## Background information on regression testing
Regression Testing makes sure there are no rollbacks in quality. It uses a pre-existing set of tests to acheive this.Testing is time constrained problem dealt with by proper selection and prioritization. During selection the goal is to select only those tests exercising the code directly or indirectly affected by changes. Prioritization reorders the entire test suite so that tests with higher priority are run first. They can be combined, by selecting a subset of tests, then prioritizing them, or by prioritizing the suite, then selecting tests.
## ML for selection and prioritization
Learning to Rank is a supervised learning where the algorithm is trained on a labeled dataset. Training is done on a number of observations, depending on the amount of history available; the resulting model is used to prioritize tests for next commits. The model needs to be re-trained from time to time: this is preferred to online learning when training is expensive (e.g., for a large codebase).
Ranking to Learn is reinforcement learning where the agent learns by receiving feedback in the form of rewards or punishments as it interacts with its environment. Training is done online, namely when the agent updates its knowledge about state-action-reward. RTL algorithms take much longer, since training is repeated at each commit. But RTL can rank test cases since the beginning.
## Algorithms
The LTR strategy can use pointwise, pairwise or listwise algorithms Pointwise LTR. The ranking problem is transformed into classification, regression, orordinal classification, and solved with respective existing methods. The training data are typically supervised learning data; given a sample(a test target), the algorithm predicts the class label, real number or grade label for the three cases.For example, in classification problems the score can be the probability of a test belonging to a class (e.g., high-priority and low-priority, in a binary formulation); in regression and ordinal classification problems, a function of the testing objective yielding a real priority number or a grade. The loss function in learning is pointwise in the sense that it is defined on a single object (featurevector).
PairwiseLTR Ranking is transformed into a classification or regression problem, where a sample is a pair of test targets: a model can tell which test target has higher score than the other in a pair. The goal is to minimize the average number of inversions in ranking, due to unordered pairs.
ListwiseLTR. The problem is addressed in an intuitive way, as ranking lists are taken directly as samples in both learning and prediction.The approach trains a model able to assign scores to feature vectors and rank them accordingly. The goal is to minimize the difference between the predicted and the actual ranking lists. In classification-based pointwise LTR, we consider four classes derived from the two above-mentioned prioritization criteria(fault detection and execution time), which are (in decreasing priority): Class3: At least one failure is detected running the test target, and the execution time of the test target is shorter than a threshold (computed as the median of execution times on the last W samples); Class2:Atleast one failure is detected running the test target, and its execution time is longer than the threshold; Class1: No failure is detected running the test target, and its execution time is shorter than the threshold; Class0: No failure is detected running the test target, and the execution time of the test target is longer than the threshold.
This work evaluates the ten algorithms listed in the table above. They can be further classified into ensemble and non-ensemble algorithms: the former category includes: RF, RL-RF, RankBoost, MART, L-MART; the others are non-ensemble. The Weka 4 and Knime 5 tools were used for point wise algorithms, and the RankLib library for pairwise and listwise algorithms. The number of samples used for training, initially set to 2,000, is subject to sensitivity analysis.
## Experimental factors
It investigates what factors make some ML algorithms behave better than others for test prioritization in a CI context. It focuses on characteristics of the code under test and of the CI process. The code under test factors include
```
* The variability of the code/test metrics;
* The failure proneness of the code, which causes more or less balanced datasets;
* The code/test metrics that can be used as features for training and improving prediction.
```
The CI process factors include
```
* The inter-commit time, which determines the time available for performing TS&P;
* The cycle size (i.e., size of the sample, or number of tests, per commit), which affects the length of the history available for learning.
```
## Results
### RPA is the metric used to measure accuracy
MART and L-MART have the best RPA and medium-level ranking and training times. CA performs poorer but is better for ranking time with a medium-level training time. KNN requires less training time than others LTR, but it takes long for ranking, not justifiably paid off by the RPA. RL algorithms require high training times, being online learning schemes. RL-RF has high ranking times, but good RPA. RL and RL-MLP have better times but poorer RPA.
All LTR approaches are more affected by the 𝑇^2,code variability indicator. While all RL-based algorithms are more robust to variations, likely because of their online-learning nature. This suggests that in highly variable contexts, with intense metrics’ changes, RL-based approaches can be preferred, to avoid having to re-train a static LTR algorithm too often. In contrast, static contexts can stress the good performances of LTRs.
Inter-committime. RPA measured algorithms’ performance regardless of the time available to execute test cases. If time limits do not allow to run all selected tests, the tester might be interested in analyzing the algorithms’ performance under various time constraints. Based on a previous work studying regression testing under different time constraints, they consider scenarios in which 25%, 50% and 75% of the number of selected tests can be run at each cycle, and assess the impact on ranking effectiveness.
Whenever results are in line with the RPA, e.g., L-MART and MART on test execution times or RankBoost on number of failures that are close to the optimal one, it means that mis-ranked tests had low impact (e.g., because their execution times or number of failures is not much different than those of the optimal ranking’s tests). When results are not in line with RPA, e.g., RL-RF and RankNet on failures, then either the algorithm performs well for one prioritization criterion and bad for the other and/or the tests that were mis-ranked, even by a small amount, had a high impact. Clearly, short inter-commit time may influence the choice of the algorithm to adopt more than the RPA metric.
History length. They investigate whether and to what extent the amount of history used for learning impacts performance of the algorithms. For LTR, this history is used just once at the beginning for training; for RTL, the history is in a sliding window (the memory) used to update the learning process online. The RPA is quite insensitive with respect to a training sample size bigger than 500 observations,for all the algorithms. The average good performance of LTR even with smaller training sets mitigates the draw back of not having predictions during the training process. Contrarily, the training time expectedly increases remarkably with training sample size (the graph is in logarithmic scale): the RTL algorithms are severely affected,followed by pairwise and listwise, and finally, the least affected ones, the point wise algorithms.
## Conclusions
A valuable approach under short inter-commit times is to run a ML prioritization algorithm after selection, so that it acts on a small problem size and on test cases relevant for that commit. An incremental static selection approach like the one experimented with appears suited for the time requirements of CI.
The testing criteria should be identified first, along with their relative weight within the ranking function : they determine what the optimal ranking is. If a coarse -grain ranking is enough, then classification algorithms may workwell; otherwise, regression approaches better manage fine-grain ranking problems. Most of the experimented algorithms can work with both formulations.
The ML algorithm’s input consists of the features more likely related to the identified testing criteria. We targeted feature selection in an algorithm-independent way (unsupervised), first defining relevant features and then choosing the algorithm based on other requirements–this simplifies the problem. An alternative is to run feature selection for each potential algorithm being considered, so as to infer the best features for each of them: this, although more precise, can be much more time-consuming. In the plethora of ML algorithms to choose from, a suggestion is to first decide the learning strategy and, possibly, the category (ensemble or non-ensemble), and then opt for the specific algorithm. The choice depends on the requirements for prioritization, in terms of desired ranking effectiveness and efficiency, tolerable sensitivity to code features, and on the CI process features. They are strongly dependent on the CI context.
## References
Bertolino, A., Guerriero, A., Miranda, B., Pietrantuono, R., & Russo, S. (2020). Learning-to-rank vs ranking-to-learn: Strategies for regression testing in continuous integration. Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 1–12.
Majid Babaei,(2023,Nov), Lecture16-CICD Pipeline, School of Continuing Studies, McGill University
# Topic#10 \[Taken]
Topic 10: [Uncovering the benefits and challenges of continuous integration practices](https://ieeexplore.ieee.org/abstract/document/9374092/?casa_token=H_di3ZkRu8EAAAAA:DMlJXJhRcj-oXiFAIJBJzB_Ybrevi_d2t7ivneiGAOtJLZUmmJoU_IeL-Btf_Qn8epgEz0gX)
{% hint style="warning" %}
This topic is taken by **Abraham Somech, Liam Serour**, and **Samuel Vasserman**
{% endhint %}
## Introduction
The paper starts off by defining Continuous Integration (CI). CI involves iteratively integrating small changes into a codebase, supported by tools and processes for rapid feedback and quality improvement. As its name suggests, CI focuses on keeping the code and contributors integrated and on the same page. The study examines the implementation of ten core CI practices defined by Fowler and Foemmel in 2006, exploring how these practices are adopted, their advantages and disadvantages, and the challenges they pose in different organizational contexts.
## CI Practices
1. Maintaining a Single Source Repository
2. Automating the Build
3. Making Builds Self-Testing
4. Daily Commits to the Mainline
5. Ensuring Each Commit Builds on an Integration Machine
6. Keeping the Build Fast
7. Testing in a Production-like Environment
8. Easy Access to the Latest Executable
9. Visible System State and Changes
10. Automated Deployment
The paper outlines the advantages and disadvantages of each of these practices. It then discusses how these practices are widely adopted but vary in implementation depending on context and perceived benefits.
## Methodology
The methodology used in the study of Continuous Integration (CI) practices was essentially split into two categories. Four of the practices can be quantified and thus they were analyzed by data-driven analysis without needing to speak with workers at the companies. For the other practices, it was necessary to conduct interviews and speak to workers in order to understand how the practices were being adopted.
Data-Driven Analysis:
Data-driven analysis focused on CI practices that could be quantitatively measured and thus observed directly from CI artifacts. This included practices that have a direct output (visibility) in the software development process. The specific CI practices studied through data-driven analysis include: Automating the Build: This practice could be studied through data-driven analysis by examining the build logs to check if automated builds are used or not. Daily Commits to the Mainline: By looking at the trace logs, it is possible to see the commit history. However, some of the commit history was lost due to commits that were squashed for instance. Ensuring Each Commit Builds on an Integration Machine: This could be studied by looking at the repository activity to see if the commit resulted in a build by looking at the build status indicator. Keeping the Build Fast: Build logs could be used to analyze the duration of each build, assessing whether the builds are being kept fast.
Interviews for Understanding Adoption:
The other CI practices cannot be studied from data alone. For instance, determining if changes have visible state changes is something that needs to be verified by speaking to workers. In order to do this, many different organizational members from the companies were interviewed. The interviews were conducted by using open-ended questions and asking the participant to elaborate on things.
## Findings
The organizations varied in their adoption and implementation of CI practices due to differences in project context, practice perceptions, and process constraints. Before delineating the main findings, the paper explains the organization profiles, which will give insight to why there were differences in implementation.
Organization Profiles:
Organization A: does data processing and deals with projects that involve large data volumes and complex processing requirements. Their CI practice implementations would reflect the need to handle these data-centric tasks efficiently.
Organization B: An online content provider, using its publishing platform, such as advertisement management. Recognized for their structured, automated workflow and strong commitment to CI practices.
Organization C: Specializing in online bookings, this organization's projects would involve integrating multiple services for a global customer base. The CI practices here would be aimed at ensuring robustness and reliability across various integrated systems.
Differences in CI Practice Implementation:
**Maintaining a Single Source Repository:**
Organization A prioritizes the reduction of merge conflicts in their development process. To achieve this, they have structured their repository by separating components based on the need for simultaneous updates. This approach is specifically designed to minimize conflicts that arise when multiple components are updated concurrently, reflecting their strategic focus on maintaining a smooth and conflict-free development workflow.
Organization B opts for a distinct approach where each component is maintained in its own separate repository. The rationale behind this structure stems from their inability to see the clear benefits of a monolithic repository, commonly known as a mono repo. However, this decision necessitates duplicating their workflow processes and maintaining complex dependency graphs between components. This structure could potentially introduce increased overhead and complexity in their development process, highlighting a trade-off in their repository management strategy.
Organization C initially adopted a single repository structure, driven by workflow considerations. However, they later transitioned to branching off different parts of their project into separate repositories. This shift was primarily due to limitations in their tooling capabilities. The tools at their disposal may not have been adequately equipped to manage a large, integrated codebase efficiently, prompting them to modify their repository structure. This evolution in their approach underscores the impact of tooling on repository management decisions and the need to adapt to technical constraints.
**Making the Build Self-Testing:**
In software engineering practice, different organizations prioritize various aspects of their development processes based on their unique perspectives and resource allocations. Organization A, for instance, emphasizes the importance of faster build times prior to the pull request (PR) stage, coupled with a strategy where lengthier, more comprehensive tests are conducted during the review phase. This approach is rooted in person 1 who expresses a preference for minimizing the duration of tests conducted on the developer's side, thereby optimizing the developers' time and efficiency. On the other hand, Organization B prioritizes financial resources over the development of their infrastructure. Organization C places a higher value on thorough testing, even if it results in longer build durations. This reflects a commitment to quality and reliability in their software products, demonstrating how organizational priorities can significantly influence the balance between speed, efficiency, and thoroughness in software development methodologies.
**Keeping the Build Fast:**
The duration of builds is a critical factor that varies significantly across different organizations. This is influenced by their specific priorities and operational contexts.
Organization A aims to keep most of their builds quick. They recognize the advantage of reducing the problem of context switching for developers. Rapid builds enable developers to stay focused on their current task without significant interruptions. However, some builds in this organization may take longer, particularly in scenarios involving extensive data processing. This indicates a nuanced approach where the build duration is adapted based on the specific needs and context of the project, balancing speed with the demands of complex tasks.
Organization B has streamlined their process to ensure that builds take no longer than 10 minutes. This brings two key benefits: it minimizes context switching, allowing developers to maintain a high level of concentration and efficiency, and it reduces blocking in the development process, ensuring that team members are not left idle waiting for builds to complete. This approach reflects a strategic emphasis on maintaining a high tempo in development cycles and optimizing developer productivity.
Organization C experiences build durations ranging from 10 to 20 minutes. This longer build time is influenced by two main factors: a higher priority placed on thorough testing and certain infrastructure limitations. The organization’s decision to prioritize comprehensive testing over build speed underscores a commitment to quality and reliability, even at the expense of longer build times. The infrastructure limitations further contribute to these extended durations, indicating a potential area for future improvement or investment.
**Automating Deployment:** In the context of deployment processes, all three organizations have opted for manual triggering for deployments. However, each company had a distinct approach in doing so that reflects their culture and hierarchy. Organizations A and C reserve this privilege for senior members, ensuring that deployment decisions are made by experienced personnel, preventing indiscriminate deployment by less experienced team members. This approach emphasizes a controlled and hierarchical decision-making process. In contrast, Organization B adopts a different approach, assigning the responsibility of deployment to the developer who authored the change. This strategy is rooted in the rationale of fostering a culture of ownership, where developers are directly accountable for the changes they implement. This approach encourages developers to be more thorough and responsible, as they are directly involved in the deployment of their own work, promoting a sense of ownership and responsibility within the team.
## Discussion:
The study found that variation in implementation practices of CI practices are related to the following themes: differences related to project context, practice perception and CI-related process constraints. The context of the project clearly has an impact as we saw. For instance, if there is a lot of data for testing, then of course the tests will take longer. The practice perception also had a large impact. Some organizations viewed testing as detrimental when it is too long and made a decision to keep it short due to that. Regarding CI related process constraints, it is crucial to keep in mind that the goal of CI is to speed up the development feedback cycle. However, integrating CI practices can sometimes lead to increased cycle times. For instance, waiting for other pull requests (PRs) to be completed before proceeding with merging. This ensures better integration and consistency but it also slow down the cycle by blocking merges if there is a backlog of PR’s. These variations challenge the notion of CI as a uniform set of practices, and the need for a nuanced understanding of CI in different contexts to apply them properly.
## Conclusion
The study concludes that CI is not something black on white where we can definitely say whether a company follows it or not. Instead, CI should be viewed as a spectrum that must be tailored to specific organizational contexts and needs. The degree to which CI is adopted is influenced by various factors such as project type, testing strategy, and organizational perception. Understanding these nuances is crucial for both researchers and practitioners to understand why CI is being used in a particular way in a company and how it should be used.
## References
Elazhary, O., Werner, C., Li, Z. S., Lowlind, D., Ernst, N., & Storey, M.-A. (2021, March 7). Uncovering the benefits and challenges of continuous integration practices. arXiv.org. M. Fowler and M. Foemmel, “Continuous integration,” 2006, Accessed: May 21, 2020. \[Online]. Available: . com/ycbl2uhj
M. Fowler and M. Foemmel, “Continuous integration,” 2006, Accessed: May 21, 2020. \[Online]. Available: . com/y8d3asjv
M. Heller, “Continuous integration: The answer to life, the uni- verse, and everything?,” 2015, Accessed: Jul. 14, 2020. \[Online]. Available:
D. Stahl and J. Bosch, “Experienced benefits of continuous integration in industry software product development: A case study,” in Proc. 12th IASTED Int. Conf. Softw. Eng., 2013, pp. 736–743.
# Lecture#1: INT
What You Need to Know!
{% hint style="danger" %}
There is no recording for this lecture!
{% endhint %}
{% file src="/files/vzxEz4cjIPtPv8pE7a1m" %}
# Lecture#2: PLN
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EfUnOEsY4xZNsI9A92oUffkB9MluHdgQ92hVtR5ai7GQxg?e=SkEpeZ)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/AqM96LEo7qVr5oLBdhmc" %}
# Lecture#3: Git 1
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EWS5gb6kFWhCtQ_9tHBGD7MBGzWdEvpJmIQ1e94tofDWMA?e=WKKw4i)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/ig9VExWu2bz6OX92YfM6" %}
# Lecture#4: Git 2
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EdAs0k9jY_VKtNewluH2TacBpYqLejN8K0QULDUbmXrF5A?e=OqIuas)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/7zkn3WA1wqO5GthBK43J" %}
# Lecture#5: Git 3
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EV5tn9h_zj9EhC4l1BSCWW0B6LGKGx8kcb6TWk7Bfy3hdA?e=dCXsp7)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/kiZ7oQbMwa8lm9pbqqWP" %}
# Lecture#6: Git 4
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EeQczSuo6Z9Ho01S1GTG_okBAvi9qTrpmDM0OmSVTYi9lA?e=3uRPe5)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/asmwB2qfQZZdQuQHvllF" %}
# Lecture#7: CR 1
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EVM-JCQex-9DuTQBKlXyzs0BDK_U8QPq4my93Di8_XW5cA?e=NHB1by)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/JobBxTLbsKKM1hNKl3oz" %}
# Lecture#8: CR 2
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EWLqJNxI39JIlG7dMaya-8IBZwYmg2s3GResv6hGZCHzRw?e=G5jlqE)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/TfxXjC9DFjzHEhua4C8R" %}
# Lecture#9: CR 3
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EbxvtC9A3DRAlP9Qk5qQdEEBfXVZRv1rataygqn75gXY_A?e=fx28E9)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/S8RkqBRoo4d7daOX7Qjj" %}
# Lecture#10: DOK 1
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/Ec-gsDZntMNGpRnl7IdkrtMBYF4P5hivVog0qIg8sGukeQ?e=0suNnq)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/UOxdErvC0jzUo5HUdhiU" %}
# Lecture#11: DOK 2
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/ETZV_7wZfIFEsnZpQn4YIB0BrFhViiTMc_ROV4DYRpD-iQ?e=BbpheL)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/km3AKEueSppY1L2Fuc06" %}
# Lecture#12: DOK 3
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EYpcpME90xhOqZXzqlu5HLwBTijmvLaHL6DBgHe4lhFT_g?e=qUeE34)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/c7DzdM95dJmzhfNOUVH9" %}
# Lecture#13: DOK 4
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EVmUuzv_9HVBt1nOINoYLaEBLivGc72YXmOW78sk2yyWfA?e=1YcK5W)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/zPEYZTBc2adNfkROgnaa" %}
# Lecture#14: DOK 5
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EVdcElv6NzxBqdYH7uDOiPQB8lV3QV1La_WTd6stJPeYSA?nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0RpcmVjdCJ9fQ\&e=c8Mu0j)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/o76pxyPPIKQm4hK7u3pA" %}
# Lecture#15: CICD 1
What You Need to Know!
{% hint style="info" %}
The recording can be found [here](https://mcgill-my.sharepoint.com/:v:/g/personal/majid_babaei_mcgill_ca/EZ7TKEiZ0xVDsiTlh2VUB7sB3ROrHDiSUeC51QQatYLGtw?nav=eyJyZWZlcnJhbEluZm8iOnsicmVmZXJyYWxBcHAiOiJPbmVEcml2ZUZvckJ1c2luZXNzIiwicmVmZXJyYWxBcHBQbGF0Zm9ybSI6IldlYiIsInJlZmVycmFsTW9kZSI6InZpZXciLCJyZWZlcnJhbFZpZXciOiJNeUZpbGVzTGlua0RpcmVjdCJ9fQ\&e=DhKMNs)!
(*only available for students at McGill University, others can get access upon my premission*)
{% endhint %}
{% file src="/files/pQFGfaKG2uXzubOgODVp" %}
# Lecture#16: CICD 2
What You Need to Know!
{% hint style="warning" %}
There is no recordings for this session!
{% endhint %}
# Plagiarism and Cheating
What You Need to Know!
{% hint style="info" %}
According to the rules and regulations in the [**Code of Student Conduct and Disciplinary Procedures**](https://www.mcgill.ca/students/srr/academicrights/integrity/cheating)**:**
{% endhint %}
**16. Plagiarism**
“Plagiarism” means the representation of another’s work as one’s own or assisting another in representing another’s work, published or unpublished, as their own.
**(a)** No student shall represent another person’s work, published or unpublished, as their own in any writing, such as an essay, thesis, research report, project or assignment submitted in a course or a program of study, or represent as their own the work of another, whether the material so represented constitutes a part or the entirety of the work submitted.
**(b)** No student shall contribute any work to another student with the knowledge that the latter may submit the work in part or whole as their own. Receipt of payment or other forms of compensation for work contributed shall be cause for presumption that the student had such knowledge.\
\
**17. Cheating**
No student shall:
**(a)** In the context of an Assessment, obtain or attempt to obtain information from another student or an unauthorized material including from an electronic device or give or attempt to give information to another student or possess, use or attempt to use from any unauthorized material including an electronic device;
**(b)** In the context of an Assessment, remove, retain or alter examination or Assessment materials.
**(c)** Represent or attempt to represent oneself as another or have or attempt to have oneself represented by another when completing an assessment;
**(d)** Submit in any course or program of study, without both the knowledge and approval of the person to whom it is submitted, all or a portion of any writing, essay, thesis, research report, project or assignment for which credit has previously been obtained by such student, or which has been or is being published by such student, or which has been or is being submitted in another course or program of study by such student, at the University or elsewhere;
**(e)** Submit in any course or program of study any writing, essay, thesis, research report, project or assignment containing a statement of fact known by the student to be false or a reference or source that they know has been fabricated.\
**18. Confidential Materials**
**(a)** It shall be an offence knowingly to procure, distribute, or receive, by any means whatsoever, any confidential academic material such as pending examinations or laboratory results or other instructor-generated materials or documents without prior and express consent of the instructor.
**(b)** It shall be an offence knowingly to distribute, or use for any reason other than one’s own private study, by any means whatsoever, any copyrighted material or material owned by another person.
**19. Misrepresentation of Facts**
**(a)** It shall be an offence to knowingly misrepresent material facts to another for the purpose of gaining admission to the University or obtaining academic advantage or credit.
**(b)** No student shall in the course of their academic life knowingly defraud or abuse the trust of any University office, facility, or service.
Policy on Text-matching Software:\
McGill is committed to promoting the highest levels of academic integrity, which is fundamental to achieving our mission of the advancement of learning. In promoting academic integrity, McGill strives to provide information about the meaning of integrity, about how to foster it, and about the consequences of breaching it.\
\
*You can review the Code of Student Conduct in full, and other relevant policies* [*here*](https://www.mcgill.ca/secretariat/policies-and-regulations)*.*