Testing is an essential part of any software development project, but it can be especially challenging when dealing with long-running projects that involve multiple components, and technologies. Such projects may last 10+ years so within a time they went through many different challenges like a requirement to manage a huge amount of functionality, changes in the team within a time, missed expertise on some points, migration to other technology, a requirement to support different versions and platforms, etc. So as a challenge for the development team, it’s going to be a challenge for the QA team to successfully deal with all the challenges and help to make the project successful.
In this article, we will define the main challenges and offer solutions while testing on long-running projects. Elinext offers QA testing almost since its foundation in 1997 and we are glad to share our experience with you!
First, let’s try to define typical problems that may be faced in the testing process on such projects:
- Test maintenance: Test maintenance is the process of test case management to keep them up to date with the current project requirements. Test maintenance can be challenging and time-consuming, especially for large and complex projects with frequent changes. The longer a project lives, the more functionality it includes, and, as a result, a significant number of test cases are created. But within a time, the test cases may become outdated, irrelevant, or redundant. This can lead to false positives, false negatives, or missed defects.
- Test coverage: Test coverage is the measure of how well the test cases and scripts cover the functionality, features, and requirements of the project. Test coverage can be affected by factors such as the size and complexity of the project, the availability of resources and time, the quality of the requirements and specifications, and the testing strategy and approach. Low test coverage may lead to undetected defects or failed deliveries.
- Environment: The test environment is the set of hardware, software, network, data, and configuration that is used to execute the test cases and scripts. It can be affected by factors such as the compatibility and interoperability of the components, the stability and reliability of the network, the availability and security of the data, and the consistency and accuracy of the configuration. With an inappropriate test environment, you will be spending much more time on the investigation of inaccurate test results, wasting time on unnecessary fixes and investigations, and test maintenance.
- Communication: Communication is the exchange of information among the testing team members and other stakeholders involved in the project. Such items as the size and location of the team, communication channels and tools, and the frequency of communication are directly affected by communication effectiveness. With poor communication, you have to spend additional time for clarification of misunderstandings, increased chances for conflicts, and missing important information that can affect your possibility of delivering requirements in time.
- Immediately not visible benefits: many decisions that need to be made at the beginning of a project and followed up later lead to additional costs, while the benefits from them are not immediately visible. Such items like documentation creation, code review, and unit tests, not checklists but test cases require constant, additional costs and time, slow down the development process, and, at the same time, the profit from them you will get only within a time.
- Lack of information: over time, knowledge of the logic of particular modules or system components may be lost. This may be due to missed documentation or the loss of key employees. To restore this knowledge, you need to either review the logic in the code or investigate through the interface but both ways may not be easy.
A correctly defined test policy, which includes considerations for manual testing services, may prevent many problems or, at least, minimize the negative impact of the items listed above. A test policy is a plan that defines the scope, objectives, methods, and tools of testing for a project. A good test policy can help ensure the quality, reliability, and usability of the software product, as well as reduce the risks, costs, and time of testing. Defining policy for complex projects is a complex task, which requires a deep understanding of the project goal, requirements, architecture, and design, as well as the testing purposes, available resources, environment, and restrictions. In the scope of this article, I would like to discuss some of the key aspects of defining a test policy for such projects and share some tips based on our experience.
Test policy
The project should be started with at least some initial criteria defined in the test policy. Within a time, it will be extended and updated according to the actual project state. To define a test policy, you need to follow some steps and include some elements in your document.
First, understand the purpose and scope of your test policy.
A test policy is a document that describes in overall words the goals, principles, and processes of testing for the project. It should align with your quality policy and vision, and reflect your organizational culture and values. A test policy should also apply to both development and maintenance projects, and cover all types of testing that you perform.
Second, describe the benefits you will get from the testing for the project. In the scope of this part, you need to explain the profit value to your customers received from the effective project testing, as well as improve the testing effectiveness of your software products or systems. You should also justify the cost of quality and testing, and show how testing can reduce the risks and costs of defects, failures, and rework. Btw, review the documentation on a regular schedule and make it actual – it’s a good practice on any project.
Third, define your test objectives and expectations. You should specify the goals that you want to achieve, such as ensuring quality, finding issues, verifying requirements, etc. Also define the metrics that you use to evaluate the effectiveness of testing, such as test coverage, defects lifetime, feature delivery time, etc
A very urgent item in this chapter is to correctly specify acceptance criteria – the conditions that the software product must meet to be accepted by the customer. Based on experience, we can say that there are some dependencies on the development cycle length. The relations between the length of the development cycle and acceptance criteria are complex and depend on various factors, such as the project scope, size, complexity, quality, methodology, and resources. However, some general trends can be observed:
– A shorter development cycle can lead to more frequent feedback and validation from the customer or the user, which can help refine and adjust the acceptance criteria as the project progresses. This can result in a better alignment between the software product and the customer or user expectations, as well as a higher satisfaction and acceptance rate.
– A longer development cycle can allow more time for planning, designing, and implementing automated software testing services, as well as reviewing the software product, which can help ensure that it meets the acceptance criteria defined at the beginning of the project. This can result in a higher quality and reliability of the software product, as well as a lower defect rate.
– However, both a shorter and a longer development cycle have their drawbacks and challenges. A shorter development cycle can increase the pressure and workload on the development team, which can affect their performance and motivation. It can also increase the risk of scope creep, technical debt, and rework. A longer development cycle can increase the cost and complexity of the project, which can affect its feasibility and viability. It can also increase the risk of changing requirements, outdated technologies, and customer or user dissatisfaction.
You should also describe how you monitor and report the progress and results of testing. There are different possible ways of monitoring the results of testing, depending on the type, scope, and purpose of the testing. Some of the common methods are:
- Store testing information and results: This involves keeping track of the test cases, test data, test scripts, test results, defects, and other relevant information in a centralized and accessible location. This can help ensure the consistency, completeness, and accuracy of the testing process and outcomes. It can also facilitate the analysis and reporting of the testing results.
- Create test execution reports: This involves generating and presenting the summary and details of the test execution results, such as the number of test cases executed, passed, failed, skipped, or blocked; the defects found and fixed; the test coverage; the performance metrics; and the recommendations for improvement. Test execution reports should be clear, concise, accurate, and timely. They should also provide relevant insights and evidence for decision-making.
- Provide visibility by tracking KPIs: This involves measuring and monitoring the key performance indicators (KPIs) of the testing process and outcomes, such as defect density, defect resolution time, defect severity distribution, defect root cause analysis, test efficiency, test effectiveness, and customer satisfaction. KPIs should be aligned with the testing objectives and goals. They should also be communicated and shared with the relevant stakeholders.
Fourth, describe your testing process and methods. Provide an overview of the main phases and activities of testing, such as planning, analysis, design, implementation, execution, evaluation, closure, etc. You should also explain the roles and responsibilities of the testing team and other stakeholders involved in testing. Specify the testing methods and techniques that you use for different types of testing, such as unit testing, integration testing, system testing, user acceptance testing, functional testing, non-functional testing, etc. Explain how each method can help achieve the testing objectives and overcome the challenges of complexity.
The tools and frameworks that you use to support your testing methods and techniques also should be specified here. This is a very important part of the policy as in the long term correctly selected tools can prevent many problems or, at least, they will make your process clearer and more convenient for you. Make sure that it’s possible to integrate selected tools.
Along with this part, you can also think about the following items:
- Design test cases: This involves defining the expected inputs, outputs, and behaviors of the software under test, as well as the criteria for verifying and validating the test results. Test cases should be clear, concise, specific, and measurable. They should also cover different scenarios, such as positive, negative, boundary, and edge cases. It becomes much more critical for you to have such a thing defined when you have a big QA team, because each QA has their own experience and style of test case creation, so without clear and consistent rules over time, you will end up with a mess of diverse and incompatible styles and approaches.
- Manage testing environments: This involves setting up and maintaining the hardware, software, network, data, and configuration settings that are required to execute the test cases. Test case environments should be isolated from other environments, such as development or production. They should also be consistent and reproducible across different test runs.
- Enable data sharing between team members and across different teams: This involves facilitating collaboration and communication among the testing team members and with other teams involved in the software development project, such as developers, analysts, managers, customers, and users. Data sharing can help improve the quality and efficiency of the testing process and outcomes. Based on this practice, the team can get a culture of learning and feedback. It is going to be a critical item in case several teams are working on product development, so along with data sharing, the responsibility areas between the teams should also be defined.
Fifth, as I already mentioned above, you need to review and update your test policy regularly. Be sure that your test policy is consistent with your quality policy and vision, as well as with the current needs and expectations of your customers, stakeholders, and users. You should also check if your test policy is aligned with the best practices and standards in the industry or domain in which you operate. You should also solicit feedback from your testing team and other stakeholders on how to improve your test policy.
Practical Tips
In this paragraph, I want to focus on some practical aspects that will help you to set up the development and testing process for the project. I said development, yes, because these two processes are going very close to each other and it is better to review them together.
- Everything starts with requirements. In our practice, we usually review requirements before planning and preparing questions for clarification. After that requirements should be updated accordingly to answers. This is a very important point that is often missed and, later, can lead to confusion – there will be one piece of information in the requirements, but the actual implementation is different.
- Use structure for the tickets. Ideally, after planning is done, for each new feature or improvement you should have a list of subtasks for UX mockups, implementation (UI and BE), verification(test cases creation and testing), code review (UI and BE), unit tests, and functional specifications creation. I do understand that code review, unit tests, and functional specifications – it is always “not required” or postponed by the better times. But on the long distance, they are urgent and bring a lot of benefits in quality assurance on the project. So all the development will be done in the scope of the subtasks. Any bugs, improvements, and other follow-up items, not related to the implementation of original requirements should be linked to the parent ticket. Following way you will have a visible history of the implementation of every new feature and will be able to quickly find what bugs were found during the testing and what was improved to the original requirements.
- Integrate and make things visible. Integrate all the tools you are using in the development and testing process. For example, it is very convenient when in Jira you can see what test cases are created for a particular feature and the result of the last execution of these cases. Or when you are reviewing the failed case in TestRail you can see the related implementation ticket and from there you can easily navigate to the related functional specifications. Or when a commit is performed it triggers the build process and unit tests execution then. Such kinds of things greatly save time and nerves for the whole team.
- Automate. Automation is a very urgent part of the long running projects especially when we are talking about the release time. In case when you need to execute several thousands of cases during the regression testing, or when you need to release a patch in the short term and once again repeat all those thousands of cases – automation is the key. But together with a great profit it brings additional challenges to the process organization. You should be ready to resolve many many ongoing problems during the automation like finding additional servers to run it, delivering all required features there, defining the automation team size and the scope of test cases for automation and prioritizing this scope, supporting those huge amounts of automated cases, etc but it is worth it.
- Teach the team and keep processes. All these defined cool things are useless if the team doesn’t know them or doesn’t follow them. So you should have a process of team learning on these processes and monitoring that the team is following them. It may take time, and during this time things may be improved or changed, it is a two-way process but if you decide to go this way – you need to keep it. Software development is not just a matter of writing code, but also of following a systematic and disciplined approach that ensures the quality, efficiency, and reliability of the software. By teaching the team and keeping processes, you can:
- Avoid or reduce errors, defects, and failures that can cause harm, loss, or dissatisfaction to the users, customers, or stakeholders of the software.
- Save time, money, and resources by preventing rework, waste, or inefficiency in the software development cycle.
- Improve collaboration, communication, and coordination among the team members and other parties involved in the software project.
- Enhance innovation, creativity, and productivity by applying best practices, standards, and methodologies that suit the needs and goals of the software project.
- Achieve customer satisfaction and business success by delivering software that meets or exceeds the expectations and requirements of the market.
- Monitor and adapt. Monitoring and evaluating your progress and performance about the change can help you to identify your strengths and weaknesses, as well as any gaps or areas for improvement. It can also help you to celebrate your achievements and recognize your efforts. Adopting the process to the changed conditions can help you to improve your performance, efficiency, and quality of work. It can also help you to cope with uncertainty, ambiguity, and complexity that may arise from change. However, adapting the process to the changed conditions can also be challenging and stressful, as it may require you to learn new skills, change your habits, or deal with resistance or conflict. But whenever you make changes you need to understand the reasons and benefits of the change. Sharing your thoughts, feelings, and concerns about the change with your colleagues, managers, or other stakeholders can help you gain support, feedback, and guidance. It can also help you to resolve any issues or conflicts that may arise from the change.
Conclusion
So, in this article, we reviewed typical problems in the testing process on a complex, long-running project. Such items like – maintenance of a constantly growing amount of test cases, insufficient test coverage due to blurred responsibility areas and overlapping functionality, complicated communication due to the presence of several teams working together, difficulties in setting up a test environment may lead to failed releases, insufficient product quality, missed issue and difficulties in further development and support of the product.
Correctly defined and maintained testing policy allows for to prevention of many of these issues. Starting with a basic rule definition, the testing policy should evolve with the project and define the actions needed to successfully overcome or prevent emerging challenges. The testing policy should be properly integrated into the development process and complement it, however, this statement is also true in the opposite direction.
Not all of the above solutions will be useful on every project. Each project is individual and requires a separate analysis and selection of components necessary to achieve the required product quality.
