Summary
In the following post we’ll look at attempting to identify the role a tester plays in a prototyping team. We’ll discuss feedback provided by the testing community and devise an initial approach that a tester could take.
Background
My workplace has decided to put a big effort on prototyping for the next couple of months. It’s something new for us as a company as we’ve always just went away and built fully functioning software that’s been asked for. We’ve never had to prototype anything before.
So certainly it’s a step out of the comfort zone for everyone involved. One which will require a lot of thought and collaboration from all parties involved.
For myself as a tester it makes me question where I belong in this team, since a traditional functional testing effort will have less importance here; something which would have took up a large amount of my time previously. I can certainly think of places where I can provide value; however there is still a lot of uncertainty in my mind as to the role we play. It’s not a traditional test effort for sure, and requires a new approach.
So what role does a test play in a team developing prototypes?
It’s something both myself and my colleague Stuart MacDonald wanted to find out. So we sat down and had a brief fifteen minute discussion on what roles we could play to provide value to the team. Jotting down a mind map of our ideas on some paper, we quickly identified some crucial areas we could provide assistance in.
Insights from the community
Now I’m sure other people out there will have a lot more experience than us when it comes to the role a tester plays in a prototype team. As such I turned to twitter for assistance asking “How does a tester fit within a prototype team?” A bunch of people responded, and the insights they provided were very helpful for me, helping me clarify thoughts in my head and introduce new ones.
Proactive Testing
Albert Gareev highlighted the need for prospective testing skills. Prospective being something I refer to as proactive testing. Certainly, proactive/prospective testing is essential here! But to what extent?
Feeding back on requirements
As testers a key role we play is gathering information, assessing it and providing feedback on it. For a prototype team we will have some form of requirement, or even a vision if you like? So thinking proactively around the testing we could do for a requirement, we could review whatever information we currently have and gather any addition information that we need, review this and provide feedback on it. Or for a vision we could do exactly the same but obviously put more emphasis on information gathering and collaboration (questioning stakeholders) in order to provide our feedback.
So now as a result the feedback, or feedback cycles we have provided will have helped stabilise scope, reduce risk, and most fundamentally assisted in determining exactly what the end goal of the prototype will be.
Collaboration and determining business expectations
Is that as far as we can go proactively? Of course not! Prototyping will evolve over time. A good approach to prototyping will involve lots of interaction with it’s stakeholders to make sure essentially that the prototype meets their expectations, or at least indicates early when expectations are not achievable.
An approach we can take as testers here is to begin thinking of the difficult questions a stakeholder might ask of a prototype, and asking them early! From doing so we reduce the risk of developers being embarrassed and the stakeholder being upset.
Our teams are being asked to produce something demonstrable, which can be picked up and demoed to customers by our demo team. As such we already have an expected outcome placed upon us from the start. So we could begin asking difficult questions around the business expectations of the prototypes such as:
- “Are you happy to invest X amount of time in throwaway but demonstrable code?”
- Do you expect anything to be reusable from this prototype?
- Are you happy to invest knowing that what is being asked might not be achievable?
- If you do expect some re-use what are the non functional requirements of this prototype.
- Are you willing to fully commit to making these a success via frequent collaboration?
Feeding back on visuals
Rob Lambert and Peter Haworth-Langford stated that testers could play key roles in providing input to designs, process flows and UI.
I agree, and although some might question if a testers role is to input on design decisions at all, I think it can be essential at times. We might make suggestions for improvement here around usability issues we’ve identified, or realise that a piece of design or in fact the overall design itself doesn’t actually provide any business benefit.
Process flows can be mapped out early from requirements if we have wire frames. If we don’t we can communicate with the stakeholders to determine the process flow of the proposed prototype. From assessing these flows we can identify usability concerns early, for example could the flow to point X be quicker if we combined points C and D?
UI, of course there is a lot we could feed back on early here. We could even check if our competition is doing something similar? What makes their paradigms special? How do we make ours better?
Preventing rejected systems
The following two I liked a lot:
- Anders Dinsen stated “The tester understands the objectives, but questions the means to get there because he sees the problems.”
- Jamie Cuthbertson stated that a tester’s role should be to challenge assumptions, viewing things from a different perspective and making suggestions for improvement to the product.
Anders highlights the need for a tester to think of the bigger picture. It’s a difficult job that in itself, and for a prototype we’d need to fully understand what the business expectations of the prototype are first before we can begin to see if we are on the correct path.
Jamie correctly highlights the need to challenge assumptions. I’d like to add to this “assumptions that matter”. Certainly for a prototype we will be looking at things from a different perspective. I’m still unsure myself what perspective we’ll take, but for now I see myself and my colleague Stuart being focused on making sure that the end delivery satisfies business needs as opposed to individuals visions, or worse still conflicting visions. After all it’s easy for a prototype to free-fall into something which sounds and looks pretty but delivers no business value as it doesn’t actually solve a problem, or one that’s worth solving. Peter Haworth-Langford highlighted on this when he said that we should be making sure this is actually something someone wants.
Peter Haworth-Langford also added that a tester’s role might be to help a customer or stakeholder formulate their concept. I think that’s a very good point. We could collaborate with them to generate scenarios, persona’s and solid use cases around their proposals. This in turn would help identify if their was a solid business case for the prototype after all.
Functional test involvement?
Basim Baassiri highlighted that testers could provide feedback by doing exploratory tests. Stephen Hill said something similar when he stated that they could look for areas where the prototype might break. I agree to an extent here, and if the prototype is being demonstrated as its being developed we could provide assistance here to avoid embarrassment during the demonstrations. Of course exploratory testing doesn’t just stop at functional testing, from providing feedback to the requirements we will be exploratory testing as well.
Alternative prototyping
Markus Gärtner asked a series of questions to help him understand the definition of prototyping in my company. In one question I responded that traditional functional testing would be of less importance here, to which he asked “So, how would you test a teleporter like the one from NCC-1701 D?” To which I replied that it depends on what this prototype teleporter was required to do. As a prototype it might not even need to teleport at all.
A famous modern example of prototyping without actually having a working model comes from none other than Apple, who had created several initial prototypes of a proposed iPod design out of foam. Drawing the controls on the front they used these foam models as if they were actual real working mp3 players for some time. Walking around with them and pretending to interact with the controls. This allowed them to understand if the actual look and feel could be appealing, prior to investing in a working model.
So what is the role of a tester in a prototyping team?
To get more of an understanding of the role I’ll be playing I’ve generated a mind map below.
This image has been cropped, please select it to see it in an expanded size.
Suggested approach
So the Twitter testing community provided me with a lot of help in identifying what a testers role should be in a prototyping team. As the above is just a collection of my thoughts on peoples responses I’d like to see if I can model an approach of some kind, and hopefully you can help me some more by providing feedback on that.
Step 1: Clarify business expectations
We need to quickly identify what the expectations on a prototype are from the business. Hopefully this will have been done already, but if not we’ll have to input these into the chain of command. Some example questions here might be:
- What deliverables are you expecting during or at the end of the prototyping phase?
- Are you expecting re-useable code?
- If something is not realistically achievable in the given constrains how will you react?
Step 2: Feedback on requirements or gather an understanding of visions.
Obviously as information gatherers we will identify and feedback on potential issues, gaps and risks with the information we have. Most importantly we’ll be in constant communication with the key parties involved in this prototype, keeping an eye out for conflicting information.
Step 3: Validate designs
Often a prototype can have little or no requirements. We can assist both the developers and the stakeholders here by helping (in collaboration with the stakeholder) produce real life scenarios of how a customer would use the proposed prototype. We might produce persona’s as well, to highlight the different kinds of user of a system, and in doing so validate our paradigms against these, and uncover potential usability issues.
Step 4: Determine functional test involvement if any
We might be able to assist with functional testing of a prototype prior to stakeholder demonstrations. There is a high possibility that our input here might not be required at all.
Step 5: Ask the difficult questions early
This can be two fold. We can assist developers by putting ourselves into a stakeholders / business mindset and thinking of possible questions they might ask of this prototype. If the prototype has frequent progress demonstrations (and it should have) to stakeholders, we can ask the difficult questions of them also. For example the prototype might be at risk because a clear decision has not been on a paradigm to use, we can ask why that decision has not been made and when it will be made highlighting that the project is at risk by the decision delays, and clarifying that their expectations will not be met if this progresses.
Step 6: Track expectations
As a prototype might have one or more stakeholders, combined with other people wanting to input their ideas, we’ll need to do our best to keep track of the initial expectations, and make sure if these change that everyone is aware of it and in agreement.
Step 7: Be aware that business expectations may change
Even the best laid out plans for a prototype may become obsolete overnight if the expectations of the business change. They might like the concept so much that they decide it should be part of the product, and ask that you begin building it in production standard code, which comes with a large test effort in itself. Likewise the concept might be sold to a customer and require all current prototyping efforts to be dropped to work on developing this sold concept for the customer.
Thanks for reading.
Related posts:
A prototype definitely has requirements, though they might not be documented as throroughly to help with the innovative nature of such a project.
One thing you should keep in mind, is that in the late 90s a lot of bad work has been done under the umbrella of Rapid Prototyping and Rapid Application Development. I like your list, but you might want to take a closer look into Agile Testing from Crispin and Gregory to see how a tester can help on a project where not everything works as the theory book on testing claim.
On another note, I don’t really see a big difference to testing on another project here. Find out, who matters, ask them what matters, and negotiate the degree of your involvement.
Hi Markus,
Thanks for taking the out to comment.
We have requirements and visions. Visions being a type of requirement in itself, although very high level with low levels of business justification and usage examples around it. Either way I believe it’s our role to assist in fleshing out these requirements into something more realistic and achievable that does actually provide a solution to those that matter.
When I say fleshing out the requirements it could be as simple as working with the stakeholder to develop some solid use cases around the proposed prototype, to help reduce risk of a rejected system.
The Agile Testing book is on my to-do list, I’ve got about four purchased books to get through at home already
I keep seeing it being recommended though, so I best speed up my reading efforts to get around to it 
Thanks for the heads up!
I personally feel the main difference from a traditional build is the requirement for functional testing. In our current workplace examples, they have no chance of becoming part of the product as we are not even using product code currently. Therefore my functional test involvement will be very little, other than to provide assistance at the later stages in the lead up to stakeholder demonstrations and handover of the prototypes to our demo team.
Therefore the vast majority of my time will be in a monitoring / collaborative / communication role.
It’s a pretty exciting challenge, I’m looking forward to trying something a little different this time
Cheers,
Darren.
It has happened that prototype code becomes product code over night (regardless of original intentions)
So also try to do the most important functional and non-functional testing.
Hi Rikard,
Good point, and it’s probably worth mentioning in addition to this, that we should be fully aware of the risk of this happening. E.g. how likely is the quality of this prototype ever becoming product code?
On the other hand our acceptance of this occurring without prior notice probably highlights a much larger underlying issue. One if left unattended could pose a major risk to not only this project but future projects as well.
How many last minute features on projects have you seen come back once they’ve went live with a large cluster of defects? We’ve experienced it many times in the past. Now though from addressing that acceptance of last minute feature, or major code changes problem, we have greatly reduced issues found by customers.
Further more, higher up the food chain the awareness of impacts to product quality have improved, and the “Just do it!” mentality now comes with full clarity of its consequences.
Thanks for taking the time out to comment Rikard.
Cheers,
Darren.
These are all great explanations why a tester *SHOULD* be involved during prototyping and why including a tester can be beneficial to the project. But have you also considered the possibility where you are NOT involved?
Hi Michel,
Sure there are places where we shouldn’t be involved in, or our level of involvement is lesser than during a traditional feature build.
I’ve already discussed some of in the post:
-Functional testing dependent on the expectations of the project might not be required
-Test documentation overhead is removed, or reduced
-No test plans, or very lightweight mock ups of these.
-No test cases, dependent on prototype expectations
-No automation, again dependent on prototype expectations
-Overall test involvement might be reduced, for example depending on the scope of the prototype/s it may involve a part time role.
The “ilities” e.g. testability, usability, accessibility, translatability and so on might be of less a concern. Likewise performance, scalability might also be of less concern. Again this is dependant on the definition of prototyping.
In reality the “what not to test” is dependent upon the business expectations of a prototype. Understanding and clarifying them early and this will allow you to determine your true test involvement on the project.
Thanks for prompting an expansion of the topic
Cheers,
Darren.
I missed your tweet last week (holiday). Pity. This is a subject I find exciting.
Iterative development and prototyping shouldn’t be seen as a threat that marginalises testers. It should be a great opportunity that liberates them.
Traditional testing at the tail end of a development is summative testing; what have we got here? Testing of prototypes means you can do formative testing; what can we learn so we can make this better?
It’s not just about finding faults. It’s about finding out how to do it better.
I think the greatest potential benefit is improved usability. There’s a load of interesting material about how to do early usability evaluations.
Jeff Paton has written a lot of good stuff. His website, http://www.agileproductdesign has loads. Here’s a taster.
http://www.agileproductdesign.com/writing/ieee/patton_getting_software_rite.pdf
The RITE (Rapid Iterative Testing and Evaluation) method, which he talks about in that article, came from Microsoft.
Here are some more interesting pieces by Patton.
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=13178&tth=DYN&tt=siteemail&iDyn=2
http://www.uie.com/articles/best_practices/
&
http://www.uie.com/articles/best_practices_part2
http://www.agileproductdesign.com/downloads/patton_bringing_ucd_to_agile_handouts.pdf
Larry Constantine & Lucy Lockwood have also done a lot about incorporating usability evaluations into early stages of developments. This is their site.
http://www.foruse.com/default.htm
Some of it is very formal, and probably over the top for you, but there might be useful ideas.
http://www.foruse.com/presentations/inspections.pdf
I envy you. This is how I believe testers should be working; getting involved early, asking challenging questions and shaping the product. Good luck!
Hi James,
Thanks for all the useful information. I sat down yesterday and began looking at some of it, making a start by reading articles from Jeff Paton’s site. One which I’ve took some notes from already was called “The Whole Product” http://bit.ly/gFYMub which really was a fantastic read.
I introduced usability evaluations at the tail end of the last release in my workplace. It really was a fantastic experience, and because I’d planned in dedicated time for two developers to have some of the high priority items from the feedback fixed right away, the response was fantastic. Now everyone is singing their praises, and will be something we’ll do regularly from now on. I’ll write about that though at a later date.
So essentially we’ve begun our journey with usability evaluations, taking our initial baby steps. I’ve a good few ideas around improving UX already, however I think before trying these out I’ll take advantage of the information you’ve provided and do a little research into techniques and approaches other use.
As for the importance of usability in a prototype, certainly it has high value. Depending on the prototype and the business expectations of it, usability might not be a concern. So understanding business expectations early I think is the key for successful test involvement. Certainly in our current situation, when an initial expectation is to provide a demonstrable asset, it’ll be essential.
Cheers,
Darren.
Thanks for the mention. Very interesting read!
Hi Anders,
Glad you enjoyed it! It was fun and insightful for me writing it. I certainly know a lot more now from the experience than prior to writing.
Cheers,
Darren.
So what role does your company want it’s testers play in a team developing prototypes?
Good question Tony!
Our company didn’t define an actual role for us to play here, other than highlighting that our influence would be needed, but not in your traditional test to raise bugs role, hence the investigation.
I’d say the largest influence I’ve had so far is feeding input into designs, and highlighting risks.
One of our prototypes will be generating a next gen UI. This has been a fantastic learning experience for me, and I’ve been able input some excellent ideas for it.
I’d say from the suggested approach mentioned in the post all aspects have been applied to some extent, some certainly more so than others. As James had pointed out designs, UX and usability are becoming major factors in this role. I’d also say managing expectations is playing a key part as well, certainly some stake holders can push for aspects which have no real business benefit or just don’t match the context. I try to highlight this when possible and bring things back on track.
Cheers,
Darren.