Hawthorne effect and M&E

The data collection and observations of programs/project beneficiaries for results are fundamental task of M&E professionals. The true or credible reflections from data/observation always been suspicious when reports prepared or presented or announced. For example in WASH programs, one of  the desired result would be behavior change of community in regards of hand washing. The program desired community should wash hand with soaps frequently after the use of wash room, Before and after eating. During the survey/observation, data analysis and finding presentation, the  behavior change of community could be concluded. But this M&E report encircled with suspicious when post program completion phase would not able to realize the change. This often arise the doubt on credibility of M&E as well as on presented report. This situation could be explained from the principle of Hawthorne effect. 

The Hawthorne effect is also called the observer effect. When an experiment or research is being conducted, the tendency for the subjects to alter any part of their behavior because they are aware of the fact that they are being watched, this is the Hawthorne effect.

The phenomenon of Hawthorne effect is mostly used in management to enhance the productivity of the employees. In management this effect advocates, when work of people are observed their productivity enhanced. The principle is the employee will be conscious on their work when  observed and they try to do more. Giving more attention to people brings out the best in them. The effects last till the observation. After abandoning the observation, people tends to go back to original status.

M&E requires the data collection from program/project beneficiaries to monitor and evaluate the program effects. The data from survey or observations are likely to be affected from Hawthorne effect. The beneficiaries should be on conscious mode as they are being watched and supported. There will be tendency of giving only positive response rather than being critical with program/project. Also people anticipate more program/projects would be launched in their area when pleasing program/project authorities. This makes the result  more of speculation. The same should happened to the example above. The project beneficiary either did not want to displease the project/program personnel or they abandon the behavior once community felt they are no longer observed from projects/programs.

The Hawthorne effect is challenge for M&E deliverable. The conscious behavior of  respondents on one hand should result be in exaggerated form. In another, the tendency of people behavior reversion into previous status after detaching them from observation, questions the credibility of results when observation was there.

One of the mitigation from the Hawthorne effect would be finding a way to quantify and eliminate the effect during data collection. This would be difficult task as well as challenging. The findings and conclusions could be also presented mentioning the estimation of The Hawthorne effect. The utilization of triangulation tool could also be another solution to eliminate and estimation of the effect. The indirect observation or proxy questionnaire could be designed.

Thinking from the sustainability aspect for the desired change during the program/project design would be another measure to eliminate the Hawthorne effect.

The consideration of the Hawthorne effect in M&E is practiced less. Further research and development of system to identify and measurement of Hawthorne effect should enhance the credibility of the efforts M&E professionals.

Consideration of local context in development of indicators

Defining the program indicators are essential task of M&E. Without indicators, no any measurement can be performed. If there's no measurement then management of program must be in chaos.

The quality in indicators must be ascertained to have effectiveness and credibility in measurement. The consideration of local context should be taken to enhance the quality of indicators. For example: One of outcome indicator of child nutrition program could be, 

      - Percentages of children increased in meat consumption at least once a week for nutrition enhancement in X area

Thinking straight forward, the above indicator follow the SMART standard. But if further analyzed from the local contextual perspective, it might not be so smart. If X area compromise more than 80% vegetarian population, it is unlikely the outcome measurement would represent true picture of the population. This type of measurement rise on one hand, the credibility question and on another the intention of program itself.

Without understanding the local culture and social dynamics, the developed M&E framework will be in limbo and worthless.

During the construction of indicators as far as possible, the targeted project beneficiary should also be involved or the indicators should be tested and discussed with stakeholders. This mitigate in large extent the hidden untidiness of developed indicators. 

Going back to above example, either the indicator could be omitted if program is in initial phase or could be redefined if program is in implementation or data should be dis-aggregated into vegetarian and non-vegetarian in proper manner if program is in post completion phase.

US fighting against COVID19 with informed decision

MEL always talked and advocate about informed decision. Most of time this did not happen locally as well as in projects. Many developing or under developed country like Nepal, the informed decision making in policy decision is barely practiced.

Whole world is fighting against COVID19. Now US is in epicenter of Corona virus. Nobody knows what will happened tomorrow. Efforts of every nations are going on in "Hit and trail" method. Many countries are fighting with their gut power. Many countries are in locked down.

US is fighting with informed decision and this is praiseworthy. President Donal trump has announced the death projection of his own country men/women that will be around 100,000 to 240,000 (as of March 31, 2020) based on the statistical model prepared by the experts. I wonder what if this figure was announced here in Nepal by the Government of Nepal. I am sure it will total chaos here. Further, Nepal Government and bureaucrats never accept the statistical model. They will remain under own safe job logic and throw the model in the trash bin. So, I really appreciate/ salute President Donal Trump and Americans accepting the evidence.

The following Graph is from https://covid19.healthdata.org/projections . This graph has projected the cumulative death toll with upper and lower limit estimation till August from COVID19 in US. The website has also presented other important data that facilitate the decision maker and medics to make decision and act.

I wish lower boundary be correct though I strongly wish lower boundary should be very very lower and fail this projection.

Lets see, the ruthless power of data.

Informed decision against COVID-19

The whole world is fighting against the new corona virus: COVID-19. The medical experts are also new on full understanding the characteristic of this virus. The virus related data are flooding in. The preliminary statistical analysis and inferences are emerging.

People are panicking along with the Governments are in dilemma to make evidence based policy making. The major problem of this disease is unavailability of confirmed cure though large numbers of corona patients are already cured. Till this writing (22 March 2020) as of WHO  situation report, nearly 300k persons are infected and nearly 5% of infected persons died.

Governments all over the world including Nepal are making decisions on Ad-hoc basis. Lots of noises distributed without any facts and evidences. All over the world governments are announcing lock down without knowing, how long they could have to do the same. The announced lock down period also varies. Some governments have announced for a week, some two weeks and …The announcement do not have any supported facts. It is sure the long term impact of COVID – 19 will be very costly for every nation.

According to www.worldmeters.info, 100k cases are closed. From the closed cased, the outcomes reflects 87% patients were recovered and 13% were dead. Here it is noteworthy; every infected persons do not died. The same website has mentioned, as of currently infected persons in treatment, 95% are in mild condition and 5% are in critical condition. This evidence reflects even someone test positive, it does not means he/she certainly goes in critical conditions and died ultimately.

The biggest learning is from China. According to www.cdc.gov , the website of centers for disease control and prevention of US, in its The Morbidity and Mortality weekly report published on 20 March 2020 mentions the data from China suggest (COVID-19) deaths have occurred among adults aged ≥60 years and among persons with serious underlying health conditions. Only one death occurred in age <= 19 years. In US no ICU admissions or deaths were reported among persons aged ≤19 years.

As of 4,226 cases reported in US on 16 March, age of 2,449 patients known. Among the age known patients, the age disaggregation shows 9% were from age group ≥85 years, 36% were aged 65–84 years, 17% were aged 55–64 years, 18% were 45–54 years, and 20% were aged 20–44 years. Less than 1% of hospitalizations were among persons aged ≤19 years.

Further, the Chinese Scientists have published the preliminary study report on COVID 19 cases on 2143 children in Journal of pediatrics. They have concluded Children at all ages were sensitive to COVID-19, and there was no significant gender difference. Clinical manifestations of children’s COVID-19 cases were less severe than those of adults’ patients.

After China, Italy has been epicenter for pandemic effect of corona virus. Till this writing, 59k cases recorded and among this 12k cases has been closed. The outcome of closed cases is 56% recovered and 46% death. On comparing the closed cases of Italy with China, the horrible picture is reflected in Italy. In china only 5% death occurred in closed cases. The in depth study of age statistics in the Italy cases still to be published. However, Italy is the second nation with highest population with age greater than 65. According to the news published in www.telegraph.co.uk, the death is hard hit on old age people. The median age of Italy patients is 67 years while for China it is 46 years. This fact reflects why death toll in Italy is high in comparison to China. Also, it is obvious older the age, it is equally likely to be coupled with other diseases. It is expected the mortality rate could be low in Italy if death number separated that caused solely from corona virus.

Luckily, Nepal remaining safe on spreading new Corona virus. Recently prevention measures have been adopt that is praiseworthy. It is equally necessary to question mark, is control and prevention measure going on any analyzing of evidences? Many state and non-state organizations have expended the resources for prevention and control. Such as isolation wards prepared, training delivered, awareness messages disseminated and others etc. etc. This stimulated another question, is resources expended on worthwhile stuffs?

The learning of other countries as of evidences discussed above reflects the corona virus hard hit the old age citizens. The corona virus alone do not kill the patients, when coupled with other disease accelerate the death rate. Naturally old age citizens need care more than normal aged patients. Treatment for corona virus alone may not suffice for curing the patients coupled with other diseases. Likewise, the generations of evidences if epidemic out broke is equally important. The availability of solid facts help health care experts and policy makers for inform decision with evidence. The informed decision making do not put decision maker in dilemma.  Has Nepal prepared from this perspective?

Nepal has opportunity to prepare against COVID-19 based on the evidences with intelligent analyzing of facts and figures from other nations. This ensure the efficient and effective mobilization of scare resources.

Project management to Project MEL: What has done to what has achieved

As Project Manager anybody's priority will be to make sure work is done. Is completion of work is sufficient? From the perspective of implementer, it is right to limit the responsibility with in the scope of completion of work.

Source: https://guardian.ng/features/executive-briefs/the-process-of-project-evaluations/

The development project is non profit activity. Unlike profit oriented commercial projects, the success of development projects could not be measured in terms of profit that could generate after completion of projects.

The success of development projects measured in terms of achievement of intended objectives the projects had set.

So, development project director, mangers and other stakeholders should not only focus on what your are doing and what you have done. Further it need to talk about the achievement of project in terms of development objective.

Theory of Change exhibit what you are going to achieve from the project and how you gather all the works to achieve the objective or goal of the project.

Source: https://www.toolshero.com/change-management/8-step-change-model-kotter/

Power of Survey Proved

Today, election result of Delhi (Indian capital city) legislative Assembly declared. Following my earlier post on "Power of Survey", the result proved the power of survey. The table below presents the comparison on exit poll prediction and actual result.

On analyzing the result with respect to projection, it can be seen only six seats deviated for two party. This is normal. It is hard to find any survey that can precisely present the result. Nor still any body can design the survey that can precisely give the result. Always the survey results are in margin of error.

Here's if you go to earlier post, you can see the table that presents result of exit poll conducted by five media houses. In that table, four media houses has presented the exit poll prediction in range. In this range AAP party actual result touched the upper range while BJP actual result touched the lower range.

So, during the sample size calculation, we usually jot down to two terms: "Level of confidence" and "Margin of Error". These two terms are most important part of any M&E survey. It is also hardest part to explain in layman language to general mass.

Statistically, if you say 95% Level of confidence, it means a range of values that you can be 95% certain contains the population mean.Lets reflect ourselves being layman, can I make any sense on this definition?

Describing margin of error is easy compare to level of confidence.A margin of error tells you how many percentage points your results will differ from the real population value. It is simply understood, if anybody said this survey result has 10% margin of error then that means the presented survey result will vary by plus minus 10% from real population value.

But, the absolute presentation of Margin of error without mentioning the level of confidence will be wrong and misguide the audience. So, most survey results are presented in .... result with 95% confidence level and 10% margin of error.

Again, reflecting from layman perspective, it is hard to understand confidence level. Should any layman understand, surveyor are 95% confidence in their survey. Or should understand I as a user/reader of survey report be confident on 95% level in the results.

Both the understanding are wrong. It should be if same survey carried out 100 times, .... result will be same for 95 times with ...... result shall vary by plus minus 10%.

The Delhi election exit poll results are analogous with "Level of confidence" and "Margin of Error". The survey (exit poll) carried out five times by five media houses with same pattern of results and actual result had touch the predicted upper and lower range.

Thus it can be said "Power of Survey again Proved".

Power of Survey

The Capital of India, Delhi, has observed the election for Delhi Assembly. In the large set of population, Indian media usually conduct and forecast the exit poll of every election. Most of time it used to be correct. 

I remember, in the last national election exit poll conducted by National media had correctly announced BJP will win the election with the majority. Before the election, news media were very much critical with BJP and questions the winning of BJP in that election. But turned upside down after the exit poll.

As of general obvious rule, the result of survey do not used to be exact and pin point.

The following Table presents the forecast of Delhi election by five big media. The results to be announced on coming Tuesday, 11 Feb. 2020. Lets see, this time how much power of survey will be reflected in the exit poll.

Best of luck to AAP. The party formed by a common man and had demonstrated the power of common man.

No Baseline, No Worry

MEL professionals often worried about the baseline. The worries includes baseline value, methods for baseline establishment and much more satisfying the project colleges/stakeholders. Further worries worsen when project executives did not bother to have baseline in the start of project. Should project collect baseline data mandatorily? I say "Yes" if project is in initial phase. Here, collection does not mean, it should be primary data collection. May adopt any method, but should have baseline at least in intended project result indicators. What if it is in mid-time or end phase of project? I say "No Worry".

Baseline is just a benchmark. What is the height of Mount Everest? Everybody knows it is 8848 meter. What is the baseline for the height of Mount Everest? It is a Sea which is marked as zero "0" value for to measure any height of earth surface. If sea was marked as 1000 meter, the height of Mount Everest would be 9848 meter.

Similarly, what is the freezing temperature of water? Is it 0 °C or 32 °F? Both are right. Here you should make attention on the unit of measurement. In the Celsius measurement unit the value is "0" and in Fahrenheit measurement unit the value is "32".

Definitely, the measurement of Human characteristic is complex than properties of physical materials. It can make a sense, utilizing appropriate methods and tools. How should we measure the awareness level of people? Certainly, we should face the lack in universal unit and method to measure the awareness level of people? I think it is not a problem, until you follow, M&E professional ethics and integrity. Just like, freezing fact of water do not differ whether you measure in Celsius or Fahrenheit. The difference of height between sea and peak of Mount Everest remain same whether you give "0" or "1000" value to sea level. 

The following figure is from the Linkedin post by Ann-Murray A. Brown. This is about "Three Things you can do if you did not collect baseline data." This could be mantra to lessen the worry of M&E professionals.

Data analysis explained in one shot !

Data analysis and result presentation are fundamental work of M&E. The literature on data analysis goes to many tons and it should be. We can go deeper as we discussed more. But the essence of data analysis should not be forgotten. Sometimes we lost in jumble of data set. Many times we could not able to direct/advise data analyst what to do with data.

I found following figure in LinkedIn that explained in one shot about the essence of data analysis. I supposed this could be tagged in the desk of M&E professionals.

Ethical issue in M&E and Learning

I have been taking online course in www.philanthropyu.org on "From Data Collection to data use". This is specialized course I ever found in M&E learning.

The course discuss on ethical principle in first module. This is very much mind blowing to me. I think, should it be for M&E professional and practitioner: either individual or organization.

Source: from google search

Ethics is not more than correct moral behavior. The course module one explains "Ethics are a big deal in M&E. If used properly, M&E is a tool for distinguishing truth from fiction and for creating just, positive solutions. If unethical choices are made, however, it can be difficult to know what is true. Poor, unwise, or unjust decisions can be made. Even worse, your data practices may endanger the very people you hope to help."

Ethics is not a rule neither can trial under any law in the court.It is a behavior that is anticipated by society, stakeholders or peers. Most of the time, M&E professionals have to face ethical dilemma: A choice where every option has some ethical issues.

The following are some ethical standard that should better followed:

1) Do no harm: For example-informed consent for data collection and data use. Keeping respondents data anonymous and confidential.

2) Honesty: For example- Ensuring accurate data and data analysis as well as accurate result presentation. Presenting the limitation of result and findings. Disclosing the organizational interest with the outcome.

3) Competence: From vast tools and processes one should use in which she/he is confident and skillful and use competently. The competency can be improved through continuous learning and practice.

Not following the ethical issue will result the questioning towards the credibility of M&E. The donors, stakeholders and beneficiary will loose trust on M&E. Ultimately M&E profession shall be in limbo.

Why monitoring and evaluation (M&E) matters?

What is M&E? If you google, lot of definition can be found and accustomed. The commonly agreed points are, it is about collection and analysis of information. Does it sound something like doing a job of intelligence agency? Absolutely not.

Photo credit: WASH MEL project (taken from online google search)

The many definitions about M&E found in google fails to describe the collection and analysis of information of what? I think this should be from the perspective of human dimension.

The project performed and targets are achieved, after all projects are attributed as successful. The general perception holds, if progress of project achieved then the goal of project fulfilled. This is valid only on the basis of probability: may be or may not be. Since how anyone can claim the goal or result of project achieved just by fulfilling the outputs or delivering the project deliverable?

Here's come the issue of M&E. The collection and analysis of information. The information of what? Though, the project information are essential for M&E process. The information collection from the project beneficiary side is vital one. Whether it may be baseline, mid-line or end-line. Only project goal can be realized when systematically assessed from the project beneficiary side.This involves human dimension and require statistic.

The study on characteristic and dynamism of human/society is different from physical science. It is unpredictable and most of the time unknown. Unlike elements of physical science, humans vary from each other may it be from imagination, feeling and any other. The variable characteristic and dynamism of human/society can only be summarized quantitatively or qualitatively to make a sense. But not definitely like in physical science.

If any project claimed I have reduced the poverty from my project, my organization made disaster resilient society, I question really, what are the evidences???? They answer really and claim: we have this model and done this and that. None claim I have credible evidence from my beneficiary. You can also check or research. So, that's why M&E matter, for to enhance projects' credibility on results and for to really worth the efforts and expenditures.