Multivariate Testing as a Tool for International Design

As a recent postgraduate in Human-Centred Computer Systems with a keen interest in the multicultural aspects of website design, I have been invited by Oban Conversion to contribute a piece to his blog which would put multivariate testing in a wider context, that of research techniques used in international design. Here are some of my findings…

The localization industry often requires designers to understand the preferences and habits of users rooted in a culture different to their own. For this reason, localization teams have been employing various methods and approaches to learn about these groups of users, which in turn would allow them to adjust product design to a new cultural context in its use. These methods include: using local marketing data, conducting international contextual research and applying cultural models. In the rest of this article I provide a brief summary of these approaches to understanding and modeling important cultural differences and explore to what degree they have proven to be successful. Finally, I take look at cultural multivariate testing as a new, groundbreaking tool, which can complement or in some cases even completely replace existing methods.

Marketing Data in International Design

“Marketing doesn’t provide design data” (Beyer & Holzblatt in Contextual Design)

If marketing does not provide design data, it most certainly does not provide cultural-customization data either. Although marketing segmentation tends to be very useful for designers, it is not sufficient as design data on its own. Marketing addresses issues of purchase decision making and defines groups of buyers according to their demographics and motivations to acquire the product. This is useful for design as far as identifying the target market is concerned, but further qualitative exploration of that market is required in order to find out more about the actual users’ needs and work practices, which need to be taken into consideration in the design process (Beyer & Holtzblatt, 1997).

The potential pitfalls of using marketing segmentation for driving design, particularly without verification are numerous; a frequently quoted example is of designing a piece of software for a financial director, who will purchase the system, but never use it himself delegating all computer work to his secretary. This kind of misrepresentation of user needs is even more likely to occur when designing for an unfamiliar culture.

Nevertheless, marketing data has often proven very useful in user modeling and this is the area where design and marketing come together most successfully. This approach is particularly useful in the case of international user profiling, where there is often little real user data available.

International Contextual Design

There have been cases where, due to lack of user data, the design team have conducted a contextual study abroad, such as in the case of Drey & Mrazek (1996), who carried out a study of computer use in America, Germany and France. However, even this direct method of investigation is flawed. Today there are more people that are beginning to question the universal suitability of what are essentially Western-based research methods (Yeo 2001, Chavan 2005). Cultural aspects can effect the objectivity of research and feedback. Yeo provides an example of Malaysian evaluators, who due to cultural conventions, refrained from giving any critical comments about the product they were testing:

“the participants did not want to comment negatively about the spreadsheet as this would cause the experimenter to lose face. Furthermore, given that Malaysians try to refrain from giving negative comments, the participants gave positive comments instead, to save the experimenter’s face” (Yeo, 2001).

Yeo ascribed this tendency to the collectivist nature of the Malaysian society driven by “maintenance of harmony and the preservation of face” (Hofstede 1994, quoted in Yeo 2001).
Another similar example can be found in Chavan, who emphasizes a fundamental fault with the interview technique:

“All the methods [for gathering data and testing interfaces] used in the Western world are based on the premise that participants will find it easy to articulate their thoughts and feel comfortable to say what works for them and what does not (Hall Edward). However, this assumption is heavily loaded in favor of certain cultures and against others” (2005).

Chavan gives the example of India (also a collectivist society) and the “unproductiveness” of the interview or observation techniques with individuals. Subjects felt uncomfortable when put in the spotlight and the evaluation felt like an “examination”. Chavan also observed the difficulty for the Indians to critique products.

On the basis of the above examples we can conclude that travel and interaction with native target users does not guarantee good results in an international design arena, although it is most certainly more effective than relying on demograhic statistics alone.

Cultural Models in International Design

A recent method used for international design is the application of cultural models. As an established area of study, the definition of culture by various anthropologists and sociologists varies greatly and has resulted in the development of numerous cultural models. Cultural models are sets of ‘international variables’, which can be used to compare different cultures. International variables, sometimes called ‘dimensions of culture’, can refer to various aspects of culture, including subjective information, such as values, attitudes and behaviors (Hoft 1996). An example of such a variable is a division of societies into collectivist vs. individualist (Hofstede). Another is high and low-context (Hall). Some more established models have recently found their application practiced in international software and website design for the purpose of higher cultural customization. Among these, the most prominent are the models introduced by Geert Hofstede and Edward T. Hall.

Using cultural models for website and software design has as many opponents as supporters. A significant criticism of “Hofstede’s dimensions” was that ‘users originating from the same country do not necessarily fit into the cultural mold’ (Jagne et al. 2004). This was confirmed by studies such as that conducted by Simon (2001), who used Hofstede’s method and found that within the same culture there existed significant differences in perception and preference of website features based on gender to render the cultural apects difficult to differentiate. The implications is that further variation based on age group, social background, etc. could also undermine the validity of the country-wide generalization of a study.

Bearing in mind the above criticisms, one should not disregard the importance of cultural studies for design altogether. Many recent studies in various fields suggest that users consciously and unconsciously continue to behave in a way which reflect their cultural typology (de Mooij 2001, Robbins & Stylianou 2002, Sun 2001, Singh, Zhao & Hu 2004).

Cultural Multivariate Testing

So, where does cultural multivariate testing sit in this spectrum of localization methods?

Considering its current application for website design – I believe that cultural multivariate testing promises to be the most successful method for international design.

User-centered design revolves around the notion that one should get as close to the user as possible and capture their natural online behavior with minimal interference from the tester. Both marketing data and cultural models are indirect methods and lead us to try to deduce how the user would interact with a website or application on the basis of a large collection of data representing this user. The contextual method engages with the user directly, but as we have seen in the example above, this very process of direct engagement may skew the outcome of the investigation (a bit like in quantum physics where it is impossible to observe a particle, without disturbing it, Heisenberg).

Cultural multivariate testing is a step ahead of the above methods, as it is uses a direct experimental approach. Hypothesis – assuming that we have a good idea of what to test (this is where the actual difficulty of this method lies), we create several variations of the website and let our target users “decide” which version works best for them in a specific geo-location. By doing this online through website visit statistics and measuring sales conversions, we get an instant and continuing input from thousands of visitors. This allows us to eliminate all the guesswork from the equation and let the numbers speak for themselves.

While cultural multivariate testing and experimentation remains restricted to website optimization, I believe that it presents the perfect opportunity for any multi-national business to use the Internet to learn directly from and about their users.

At the moment, I am not aware of an existing case where multivariate testing has been applied strictly for localization purposes and remain on the lookout for such an experiment.


Beyer, H., & Holtzblatt, K. (1997). Contextual design: Defining customer-centered systems. San Francisco: Morgan Kaufman

Chavan, A. L. (2005). Another culture, another method. Published in HCII2005 Conference Proceedings, Lawrence Erlbaum Associates, Inc.

De Mooij, M. (2000). The future is predictable for international marketers. Converging incomes lead to diverging consumer behaviour. London: International Marketing Review. Vol. 17. Retrieved June 20, 2006, from

Dray, S., & Mrazek, D. (1996). A day in the Life: Studying Context across Cultures. In E. M. d. Galdo & J. Nielsen (Eds.), International user interfaces (pp. 242-256). New York: John Wiley and Sons, Inc.

Hall, E. T. (1959). The Silent Language. Doubleday

Hall, E. T. (1976). Beyond Culture. New York: Doubleday

Hofstede, G. (1997). Cultures and Organizations: Software of the Mind, McGraw-Hill, New York

Hoft, N. (1996). Developing a cultural model. In Elisa M. del Galdo and Jakob Nielsen (Eds.), International User Interface. New York, New York: Wiley Computer Publishing, 41-73.

Jagne, J., Smith S. G., Duncker E., Curzon P. (2004). Cross-Cultural Interface Design Strategy Retrieved May 29, 2006, from

Robbins, S. S., & Stylianou, A. C. (2002). A study of cultural differences in global corporate websites. Journal of Computer Information Systems, 42, 3-9.

Simon, S. J. (2001). The impact of culture and gender on web sites: An empirical study. ACM SIGMIS Database. New York: ACM. 32, (1). p.18-37

Singh, N., Zhao, H., & Hu, X. (2004). Analyzing the Cultural Content of Web Sites: A Cross-National Comparison of China, India, Japan and U.S., International Marketing Review, 69-88

Sun, H. (2001). Building a culturally-competent corporate web site: An exploratory study of cultural markers in multilingual web design. ACM Special Interest Group for Design of Communications. Conference (10th. 2001). Proceedings of the 19th annual international conference on Computer documentation. New York: ACM Press. p.95-102

Yeo A. (2001). Global-Software Development Lifecycle: An Exploratory Study.CHI2001. Volume No. 3, Issue No. 1