CITE’s approach to product evaluation

Summary of Week 2 of Technology Evaluation for Global Development (edX – MITx)

Evaluation is needed due to an oversupply of products.

History of evaluation

Emerging trends in evaluation within global development include: an emphasis on quality and standards to ensure that research and monitoring evaluation is being done effectively; capacity building of developing country governments and entities to be able to evaluate, and; a focus on impact as well as process evaluations. More experimental and non-experimental approaches are being developed.

Criteria, Metrics, Weightings

To make an informed decision, consider who needs the information, which criteria are needed to make the decision, and how performance will be measured? The criteria (and sub-criteria) should be brought together in an easy-to-understand format that helps facilitate a decision.

The products chosen for evaluation are critical. Not everything has to be included but market leaders and some emerging technologies should not be left out. Design of experiments , a discipline concerned with planning experiments and analysing data, can inform the choice of products.

Scoping studies

A scoping study aims to quickly ramp up knowledge of the product family and context, narrow the scope of the evaluation, identify products that are available in the evaluation study area, identify the products that are available in the evaluation study area, and define the metrics that will be tested in the evaluation. It has 7 components: detailed product description, research questions, target users, major stakeholders, context study, use cases and criteria and metrics.

A scoping study can have a desk-based and field-based component.

International development and appropriate technology

Summary of Week 1 of Technology Evaluation for Global Development (edX – MITx)

It is important to think historically because it allows us to understand how we defined problems and solutions, and the underlying assumptions of economic development, poverty alleviation and technological change.

Throughout the years, there has been a shift from macro to micro.

After the end of World War II, the 3 modernisation goals of rapid industrialisation, democracy and  education of the poor – the big push model – led by a large public sector was seen as necessary for generating economic growth. The poor was seen as neither innovative nor entrepreneurial. During this time, evaluation methodology was based on cost-benefit analysis.

Modernisation did not work as planned. Instead, there were negative changes in the economic, political and social environment, with government seen as the problem, not the solution. There was rising pessimism about the negative consequences of technology. Cost-benefit analysis was criticised for prioritising economic growth. The study of institutional constraints emerged as a new evaluation methodology.

The alternative that emerged was a bottom up approach to meet the basic needs of the poor through small-scale appropriate technology (AT) delivered by grassroots organisations.

AT declined as an idea as its political and economic impacts were insignificant. But it re-emerged in the shift towards micro-level poverty alleviation accompanied by incentives for technological innovation. The poor were seen as credit-worthy and savvy consumers.

This has led to an oversupply of appropriate technologies for which there was very little information on product performance. Hence there is a need for the rigorous evaluation of products.


Optimising Performance

Summary of Week 4 of Human Capital Strategy for Social Enterprises (Novoed – Acumen / Hitachi Foundation)

Performance management typically includes goal setting, metrics or reliable measures, feedback, and outcomes. Feedback on how employees are delivering results should occur regularly (instead of just during performance reviews). I liked the idea of Netflix’s anonymous informal 360-degree system. Even though they moved on to signed feedback, I think it would be a good start for our NGO to get into the culture of providing feedback!

Normative Expectations and Personal Normative Beliefs

Summary of Week 2 of Social Norms, Social Change I (Coursera – University of Pennsylvania & UNICEF)

Normative Expectations

Besides empirical expectations, the second type of social expectation is expectations of what people think we should do. It is a second-order belief that are often paired with an expectation of a positive or negative sanction (e.g. shaming). Depending on the circumstances, different people will matter to our decisions. Or, they may not matter because we do not care. For instance, normative expectations do not matter for customs, descriptive norms and moral norms.

Personal normative beliefs

Personal normative beliefs are what I think I should, or people in general, should do. Normative beliefs overlap with normative expectations that the lecturer (Bicchieri) states that while normative beliefs matter to choice, normative expectations carry more weight.

Bicchieri also brings up the measurement of attitudes in surveys. Attitudes are very wide (i.e. personal normative beliefs is a narrower category) and just measuring attitudes might not provide a clear understanding of underlying beliefs and expectations. Thus she recommends making a clearer distinction in surveys.


Interdependent and Independent Behaviour, and Empirical Expectations

Summary of Week 1 of Social Norms, Social Change I (Coursera – University of Pennsylvania & UNICEF)

Interdependent and independent behaviour

When people engage in a collective behaviour, it is important to understand whether the action is interdependent. This is because interdependent actions are influenced by what a person’s reference network – the set of individuals who matter to me when I have to make a particular decision – does or think we should do.

In contrast to an interdependent action, a custom is a pattern of behaviour that individuals prefer to conform to because it meets their needs. For example, people open defecate because it requires the least amount of work – it is not conditional on how others defecate or what other people think of you when you open defecate.

Changing customs can be difficult because the alternative behaviour may require collective action that entails introducing interdependencies. Thus, understanding the interdependence of a collective behaviour helps us decide what type of intervention offers the best chance of success.

Empirical expectations

Empirical expectations refer to situations when the expectations of what other people do guide my actions. They can be:

  • Unilateral (when others don’t have expectations of me)
    • When we want to imitate someone
    • When what others do is informative (social proof)
  • Multilateral (when others also have expectations of me)
    • To coordinate with others

Empirical expectations are important to the lecturer’s (Bicchieri) definition of descriptive norms, which she describes as a pattern of behaviour that we prefer to engage in because we believe that others follow it. (This differs from the definition from social psychology.)





Summary of Week 3 of Human Capital Strategy for Social Enterprises (Novoed – Acumen / Hitachi Foundation)

Before embarking on the recruitment process, you must first determine the roles that you need to hire for. For me at this point, this is the toughest challenge! There are so many roles we could like to fill that it is difficult to prioritise the most important.

The recruitment process could be viewed in 4 stages:

  1. Scoping the job – developing a scorecard
  2. Sourcing candidates
  3. Screening and selecting candidates
  4. Closing

Scoping the job – developing a scorecard

A scorecard, rather than a job description, describes a set of outcomes and competencies that define a job well done. Sounds great on paper, but that seems too output-focused (which may create skewed incentives), and might only be appropriate for certain types of roles.

Sourcing candidates

Looking in the right place is important! Ideally, I think we would recruit internally from our pool of volunteers – assuming they are on the job market!

Screening and selecting candidates

I like the advice of selecting talent that aligns with the organisational values and that contributes to the organisation’s diversity. The recommendation to conduct trial tasks as part of the screening process is effective too! That’s where recruiting from our pool of volunteers is helpful, because we already have a sense of what they are capable of.


Recruiting is just as much about selling our organisation. As a small organisation, WISE would probably face challenges with this. And there’s a chicken and egg problem. You can’t grow the organisation without talent, but you can’t attract talent if you don’t grow. Hmm!