Mobile education builds resilience during shocks in five countries - Nature

MainMore than 2 billion people live in countries that are affected by emergencies that frequently disrupt education. Although they are frequent and disruptive, education emergencies have historically been understudied. Here we present a new database that documents just how frequent and disruptive such shocks can be. Figure 1 documents the extent of school disruption for a selected set of emergencies over the past two decades. Fig. 1: Documenting a set of education in emergencies between 2005 and 2023.The graph plots an index of the length of school closures and number of people affected by shocks that have disrupted schooling by country and year. The size of each bubble increases with the duration of school closure, the number of people affected, or both. Thicker or multiple circles represent multiple events in the same year and country. More information on the data compilation is available in the Supplementary Information. The data comprise compiled school closure information based on press releases of the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) ReliefWeb, World Vision, United Nations Children’s Fund (UNICEF), the British Broadcasting Corporation (BBC) and other local outlets. Major disruptions were also identified through The International Disaster Database (EM-DAT).Schools close for long periods during these emergencies, and learning loss can be substantial2,3,4,5. Education Cannot Wait, the United Nations’ Global Fund for education in emergencies, estimates that 222 million children experience regular schooling disruptions and are in active need of education in emergencies programmes1. In low-income countries, students lose on average 10% of the academic year owing to climate shocks alone6. Moreover, school breaks, although they are not emergencies, are another regular school disruption, and often results in substantial learning loss among low-resource households7,8. These shocks exacerbate a pre-existing learning crisis, leaving many students well behind grade-level expectations9. Fewer than half of primary-age students in low- and middle-income countries are able to read a story or perform two-digit mathematical operations10.Resilient education systems need to promote learning across settings and to withstand frequent disruptions. Ideally, interventions that promote learning in emergencies should be low-cost, simple to implement and quick to deploy. They should also yield high take-up across different geographies and implementation models and should be targeted to children with diverse educational and cultural backgrounds. Understanding which approaches can be effective across these circumstances requires multi-context, multi-model studies. However, rigorous evaluation of approaches to deliver education in emergencies remains challenging and rare, especially across contexts and with governments.Here we evaluate a set of education in emergencies programmes to promote learning during large-scale school disruptions, which affected more than 1 billion children worldwide. We conducted five randomized controlled trials, including multiple delivery models, such as NGO teacher aides and government teachers, to test scalability by government systems. Our trials were conducted across five countries: India, Kenya, Nepal, Uganda and the Philippines. In all settings, schooling was disrupted owing to the COVID-19 pandemic, and several of the countries in our study experienced some of the longest school closures in the world.We use mobile phones to provide various educational interventions to primary school children. Mobile phones provide a platform that can reach households cheaply at scale in low-resource contexts11. Whereas less than 15% of households have access to the internet in low-income countries, more than 70% have access to mobile phones12. Moreover, mobile phones enable teachers to reach students at home even when school is disrupted, providing a resilient and flexible modality to provide education during emergencies. Mobile phones are increasingly relied on to provide aid via mobile money during humanitarian crises13, yet remain underutilized to provide mobile education. One treatment included a set of SMS messages, such as numeracy content provided weekly, as well as nudges to engage in educational activities. A second treatment provided additional weekly 20 min phone call tutorials for 8 weeks. The educational pedagogy was as essential as the mobile phone platform. Phone calls covered foundational numeracy and aimed to target instruction to student learning levels via regular low-cost, high-frequency assessments. This builds on both in-person and technology-based targeted instruction approaches14,15,16,17,18,19.A proof of concept in Botswana showed that phone call tutorials were effective in promoting learning during initial pandemic school disruptions20. However, questions remain on whether this approach can be scaled across contexts and when delivered by governments—a pervasive challenge for social programmes21,22. To ensure scientifically robust findings, we need evidence across diverse settings and delivery models23. By conducting large randomized trials across five contexts as well as comparing models, such as NGO and government delivery, we contribute new evidence on effectiveness across diverse contexts and delivery models, key elements of scaling science24.This study contributes to the literature in several areas. First, we contribute to the nascent experimental literature on education in emergencies. Substantial research has been carried out in this area, but much of it has been qualitative or with small samples. One exception is a randomized trial in which an NGO provided schooling in rural areas of conflict-affected regions in Afghanistan and found large effects on learning and closing of gender gaps25. We expand this literature by evaluating alternative, scalable forms of education beyond traditional in-person schools, such as remote learning, which is the only option during many emergencies when schools close. There is a robust literature documenting the cost of school disruptions due to teacher strikes26, earthquakes2, school holidays7 and COVID-19 (refs. 4,27,28,29,30,31). However, there is less evidence on effective approaches to stem these learning losses. We contribute evidence on scalable solutions to stem learning losses. A growing literature explores the effectiveness of remote tutoring, yet the results remain mixed32,33,34,35. Without a coordinated cross-country trial it is impossible to know whether differences in effects are due to the intervention or to the setting, since both vary simultaneously. For example, a recent study33 tested a related but different intervention in Sierra Leone—using a similar mobile phone platform but deploying a different pedagogy—and found no effect of mobile phone tutoring. In their study, the tutoring call was linked to a uniform national radio programme rather than targeting instruction based on student learning week-on-week—a key difference. To assess whether the approach works across contexts we cannot simply aggregate prior studies in which both intervention and context vary—we need to hold the intervention constant, while varying sites in a coordinated multi-site trial. We conduct such a coordinated five-country trial in this study.We also contribute to a growing literature on external validity and scale. Recent examples show the extent of the scaling challenge, with many social programmes that initially worked in proof-of-concept studies no longer delivering impacts when scaled across settings or when delivered by governments22,24,36,37. In contrast to some existing studies, our results identify an education approach that appears to work well across contexts and when delivered by governments. One reason might be that we leverage a particularly scalable technology: mobile phones11,38. A second reason may be the generalizability of the underlying mechanisms tested in our study, which relate to other best practices in education. For example, tutoring has been shown to be one of the most effective, although expensive, approaches to improving learning in high-income settings39,40. The phone call tutorials provide a cheaper version of tutoring that is applicable in low- and middle-income contexts. In addition, since the phone calls are one-on-one and have frequent learning assessments, they enable targeted instruction to every child’s level, another approach shown to consistently improve learning15,16,17,19. Our study further advances the external validity and scale literature by conducting randomized trials across five countries in a literature where very few randomized controlled trials are multi-country studies. We also advance the scale literature by introducing experimental evaluation both of government relative to NGO delivery as well as randomizing those teachers who implement to evaluate effects on ongoing teacher practices. These experimental variations enable us to assess questions of effectiveness within government systems.Finally, we contribute to the global education literature, with a focus on improving learning outcomes. Over the past several decades, education enrolments have improved worldwide, yet learning has barely budged10,41,42. Growing evidence reveals that popular input-only reforms, such as general teacher training, provision of computers, or school grants, are not enough to improve learning. By contrast, approaches that improve the quality of teaching, such as teaching at the right level and structured pedagogy, can generate large improvements in learning43,44,45,46. The learning gains from the phone-based tutorials tested in this study can deliver up to 4 years of high-quality schooling per US$100 (ref. 47). These types of highly cost-effective approaches can more easily scale and help address a persistent global learning crisis in low- and middle-income contexts.Main resultsWe report results on a set of rich, harmonized data across five countries using both baseline and endline surveys. Questions included a learning assessment with the child and several additional child indicators, such as well-being, self-perceptions and non-cognitive skills. We also collected parental beliefs and behaviours. A portion of the survey was conducted with the parent, and learning outcomes were collected by directly assessing the child over the phone. A set of common questions were included in the baseline and endline surveys and across countries for comparability.We report results from large-scale randomized controlled trials across five countries: India, Kenya, Nepal, the Philippines and Uganda. In Nepal and the Philippines, we include government and NGO delivery treatment arms. All countries except India also included a comparison SMS-only treatment relative to a phone call plus SMS treatment. Randomization enables identification of causal effects of the education in emergency interventions studied. Further details on the data and experimental methods are provided in the Methods section. Further details on the contexts and interventions are provided in the Supplementary Information.Our main results are shown in Fig. 2 and show positive effects for the combined phone and SMS treatment in all countries, with increasing effectiveness over time as the trials progressed; for the SMS-only treatment, effects are more mixed across countries. Effects are further elaborated on in Extended Data Table 1, where we report effects across contexts on the number of operations gained and the highest proficiency attained. We also show pooled average effects across all five countries. Column 1 in Extended Data Table 2 shows that for the combined phone and SMS group, there was a 0.321 s.d. (P < 0.001) increase in the average numerical operation across all 5 countries. SMS effects are smaller on average, with a 0.078 s.d. effect (P = 0.006). As shown in Fig. 2 and Extended Data Table 1, SMS messages only have a detectable statistically significant effect in a subset of contexts (Uganda and the Philippines). These results suggest that SMS messages can work in contexts with the largest need. However, live phone calls may be necessary to strike the balance of being intensive enough to deliver impact that can be sustained and scaled across diverse contexts while remaining cheap and scalable.Fig. 2: Main learning results across countries and treatments.Average raw unit operational learning levels gained in treatment groups relative to control groups. Treatment effects on learning outcomes are reported by country and treatment group with 90% confidence intervals within each country. Countries are ordered by the timing of the study, with Kenya first and Uganda last. The curve represents a fractional polynomial fit (phone arm) and a linear fit (SMS arm) of average learning levels gained against the order of the country in the study, each with a 90% confidence band. ‘Learning levels gained’ refers to the number of operations gained, with each operation scored as an additional level gain (beginner, 0; addition, 1; subtraction, 2; multiplication, 3; division, 4). Data are presented as treatment effects relative to the control group with ±90% confidence intervals. Sample sizes: India (n = 708), Kenya (n = 1,985), Nepal (n = 2,678), the Philippines (n = 2,469) and Uganda (n = 1,308). Further details on related results are presented in Extended Data Table 1.In addition, although effects of phone call tutorials are consistently positive, they vary in magnitude. Results are largest in later trials and in countries that experienced the longest school closures: Uganda and the Philippines. As shown in Extended Data Table 1, students in Uganda gain more than a full operation level (1.193 levels, P < 0.001). This translates to substantial proficiency gains, with 34.4 percentage point gains (P < 0.001) on the highest proficiency level. In the Philippines phone tutorials yield 0.56 level gains (P < 0.001) and a 13 percentage point increase (P < 0.001) in students reaching the highest operation level. In both countries, students were out of school for almost 2 years and the counterfactual for the control group was very limited schooling. These countries were also the later trials, with implementation fidelity improving over time. We discuss both of these mechanisms in more depth later on. Results are robust to the inclusion of a suite of control variables.Mobile phone calls saw high take-up and high engagement throughout the programme, consistent with high impact. Detailed monitoring data shows high week-on-week engagement across sites ranging from 70% to 80%, as shown in Extended Data Fig. 1, even amid disruptions and across diverse low-resource settings. This highlights that the programme reached households using a platform that they found easy and convenient to access on a regular basis. This contrasts with the low rates of take-up of other platforms, with few households typically taking up alternative delivery modes such as online, television or radio educational resources.For both the SMS group and the combined phone and SMS group, the magnitude of the impact and statistical significance are robust to different estimation approaches. In Extended Data Table 2, we include baseline controls and country and grade fixed effects (column 2), and weight results by country arm (columns 3 and 4). We also adjust for multiple hypothesis testing in Supplementary Table A5, including sharpened q values to address the false discovery rate48. Since we have one outcome and two treatment arms, we consider two joint null hypotheses simultaneously. Estimates remain consistent.Extended Data Table 3 and Supplementary Table A7 show learning outcomes across several proficiencies. Results show that the combined phone and SMS arm increases the share of students who get division problems correct by 13.5 percentage points (P < 0.001)—a 92% increase in division (from a control mean of 14.6%). These results by proficiency demonstrate that learning improved substantially in absolute terms, in addition to s.d. gains, and across a range of proficiencies. We also find increases in the share of students who are able to perform higher-order competencies, such as fractions (6.4 percentage point gains, P < 0.001). Since these competencies were not directly taught during the intervention, this reveals learning extended beyond familiarity with the content taught. These results further reveal potential dynamic complementarities in skill formation, showing that benefits of learning basic numeracy accrue to learning additional higher-order skills49.These results are large relative to the effectiveness of a typical education intervention. A recent review found that the median effective intervention yielded 0.10 s.d. gains in learning50. Moreover, a review of 150 interventions found that more than half of education interventions do not work at all47. These reviews put our results in context, with phone call tutorials being three times as effective as the median education intervention. Moreover, effects are even larger in these multi-country studies than in the Botswana proof-of-concept study, where learning improved by 0.12 s.d. (ref. 20). This result contrasts with prior literature showing that proof-of-concept studies rarely scale successfully to new contexts21 or experience diminishing returns37. Rather than finding diminishing returns, we find that results improve as the approach is adapted, scaled and tested across contexts. We explore potential explanations later in the paper, including learning from experience as well as higher need for the intervention over time.In Table 1, we show results for the subset of countries (Nepal and the Philippines) where we randomized delivery models to test scalability within government systems. Column (1) shows the pooled results: both government teachers and NGO instructors are effective at improving student learning, with government teachers improving learning by 0.40 operation levels across both contexts, and NGO teacher aides improving learning by 0.30 operation levels (P < 0.001 for both). These results demonstrate that education in emergencies programmes can be effectively implemented by government teachers, and not only by NGO implementers.Table 1 Learning outcomes for government deliveryFull size tableThese results show substantial effectiveness when delivered by governments. Prior literature has found that programmes often work when delivered by NGOs but are not replicated when scaled by governments. For example, in Kenya, contract teachers improved learning, but when delivered by the government, the programme effectiveness waned36. A large-scale government teacher training programme in Nepal similarly found no effect, largely owing to poor implementation51. Our results build on this literature, providing an alternative view: government delivery can achieve large learning gains. This is consistent with other studies that have refined interventions over time to facilitate government adoption15,17.Heterogeneous treatment effects, shown in Table 2, reveal that the programme is broadly effective for students across the distribution of baseline learning levels. This relatively even impact across students’ starting conditions, or characteristics such as baseline learning levels or gender, is consistent with the mechanism of teaching at the right level. Targeted teaching is designed so that instruction meets children where they are regardless of their grade or age or other characteristics. Thus, broad-based learning gains are likely to be due to the fact that the programme was highly and effectively targeted, benefiting children across the distribution. We also find slight heterogeneity by parental education. Column (1) in Table 2 suggests that the programme worked particularly well for students where the caregiver had lower levels of formal education (primary education or less). This suggests that results are strongest when there are fewer alternative education support systems at home. This also reveals that even in low-literacy contexts, parents can be effective conduits for quality instruction.Table 2 HeterogeneityFull size tableWe embedded an additional randomized study allocating a random subset of teachers to deliver the phone call tutorials in Nepal out of a list of hundreds of eligible government teachers. This additional experimental variation enables detection of the impact of delivering the programme on teachers’ beliefs and practices—a measure of potential spillover effects into the education system that can persist through teachers beyond the intervention. In addition, teacher randomization enables generalization to the broader set of eligible government teachers. Thus, results capture government delivery impacts which do not hinge on implementers being highly selected, and effects are likely to translate to typical government teachers in the education system.Table 3 shows that teacher practices shift; teachers are 9.3 percentage points more likely to target their feedback to students’ learning level (P < 0.05). Teachers are also more likely to get parents involved in education (P < 0.1). We further find large effects on teacher perceptions that they were able to help students learn, as well as their desire to teach, with a 15.8 percentage point gain in wanting to be a teacher if they could make the choice again (P < 0.1 and P < 0.01, respectively). Although these results are reported by teachers and should be interpreted with caution, they suggest that delivering effective programmes could unlock a virtuous cycle within government education systems, in turn motivating teachers to want to teach and to improve their teaching practice further.Table 3 Changing teacher beliefs and practicesFull size tableWe examine counterfactual learning trajectories in Extended Data Figs. 2 and 3 to contextualize gains in the treatment groups. In Uganda, where effects are largest, about 17% of grade 4 students can divide at baseline in the control group. At endline, only 10% can divide, showing substantial learning loss. In the treatment group, 48% of grade 4 students can divide at endline, fully recovering learning loss due to school closures. Moreover, only 21% of grade 5 students in the control group at baseline can divide, revealing that grade 4 students in the treatment group surpass grade 5 students in the status quo. Thus, not only does the intervention facilitate full recovery of learning loss, it exceeds typical learning trajectories by nearly an order of magnitude. In other countries, learning does not deteriorate, but simply increases slowly. The intervention accelerates learning in these contexts relative to the counterfactual learning trajectory.In the Supplementary Information, we include several additional results. We analyse effectiveness during another education emergency—a typhoon that struck the Philippines during the study. We explore additional margins of impact, including parent beliefs, student beliefs and non-cognitive skills. Finally, we explore the potential for crowd-out, to test whether tutoring time displaces alternative productive activities. Overall, we find continued effectiveness during an additional education emergency, several margins of impact beyond learning outcomes, and limited crowd-out.Mechanisms across contextsWe explore variation in effects of phone call tutorials, which worked consistently but heterogeneously, across diverse contexts. Analysing mechanisms to explain effectiveness across contexts is challenging, and should be done with caution, since multiple factors vary simultaneously. Table 4 includes a summary of key facts per country trial. We conduct exploratory analysis using suggestive descriptive data and observable systematic patterns. We focus on two mechanisms with particular variation across contexts: rates of targeted instruction and counterfactual conditions. Various additional margins vary across contexts, but do not vary systematically or consistently enough to facilitate credible comparisons. For example, in Kenya, younger students were taught but this varies only once, rather than systematically across settings.Table 4 Trial descriptionFull size tablePlatform and pedagogyIn addition to leveraging a scalable platform—mobile phones—the programme pedagogy of targeted educational instruction, often referred to as ‘teaching at the right level’, was a core dimension of implementation fidelity. Phone call tutorials were designed to target instruction to children’s learning levels. We find that variation in effect across trials is consistent with variation in targeted instruction. Although results show that phone calls were consistently effective across countries, the magnitude of effects increases in tandem with the order of the trial: Kenya, followed by Nepal, India, the Philippines and finally Uganda. This order of effectiveness tracks the degree of targeted instruction. Detailed monitoring data provides evidence of increasing implementation fidelity as the trials progress. For example, the accuracy of targeted educational instruction increases from a starting point of 50.9% of students in Nepal to 81.5% on average in Uganda52 (Extended Data Fig. 4). These data reinforce the importance of targeted instruction. As the targeting mechanism improved study after study, this coincided with the programme becoming more effective. These results suggests the ‘voltage drop’ in effectiveness over time21—all too common when scaling—is not inevitable. Equipped with monitoring data and coordinating mechanisms to learn across trials, these results show that programmes have the potential to experience a ‘learning curve’ with improved fidelity (such as having ever more targeted instruction) and thus higher impact over time as they are scaled and implemented in new contexts. In addition to the trial-by-trial improvement in targeting accuracy, we find high rates of targeting accuracy through both NGO and government delivery. This consistent targeting accuracy across implementer types may explain the high effectiveness of both NGO and government teachers. It also adds evidence regarding the importance of targeting accuracy as a mechanism for impact52.These data suggest that effective targeting pedagogy has an important role in programme effectiveness. This mechanism may explain why the approach tested worked across diverse contexts. Consistent with the importance of these mechanisms, a phone education programme that did not rely on these principles had less impact: phone calls in Sierra Leone that delivered non-targeted lessons based on a mass radio programme to students did not improve learning33. In the Sierra Leone study, the approach used an accessible platform without targeted pedagogy. To successfully scale across different contexts, ensuring that targeted instruction is in place is likely to be key to improving learning.Counterfactual conditionsExtended Data Table 4 shows several statistics characterizing counterfactual conditions. We observe clear patterns that show more severe counterfactual conditions coinciding with larger impacts. In Uganda and the Philippines, the settings with the highest impacts, we also see the most severe conditions, with more disruption and fewer alternative educational instruction options. Uganda and the Philippines have the highest degree of disruption in general and during the study, experiencing up to 2 years of disruption, with disruption occurring during the entire study period. They also have the largest share of lagging students, with 9% or less of students reaching the top operation level in the status quo. Similarly, we see that Uganda and the Philippines have low rates of outside options, with less than 22% of households saying they received any calls from educators in the past few weeks.Although these counterfactual conditions do not follow an exact rank order with effect size, the broad patterns are consistent—more severe conditions map broadly to larger effect sizes in Uganda and the Philippines. This suggests the interventions studied are likely to add value particularly in the most disrupted education in emergency settings.Cost-effectivenessAn important feature of the education in emergencies approaches tested in this study is that they have low cost. The primary tool required to implement the programme is a mobile phone, a device that most households have access to, even in low-resource settings53. Since the approach builds on existing household infrastructure, the main costs are related to content delivery and connecting with families, which are often marginal, such as airtime for phone calls. In addition to being low cost, the approach has low procurement needs, an attractive feature for governments.We carefully collected cost data in each trial. Our estimates suggest an average cost per child of the phone call and SMS tutorials of about US$11 per child. This estimate is even cheaper than estimates in the proof-of-concept study in Botswana owing to economies of scale as the approach has scaled up, including cost savings such as use of existing pedagogical material, shorter and more efficient training, and streamlined data monitoring systems. We benchmark the impacts of the programme against other education programmes using a variety of approaches. First, we compare raw estimates of effectiveness and cost with similar programmes such as tutoring and targeted instruction. Phone call tutorials simulate the benefits of one-on-one tutoring, which has been shown to be one of the most effective educational approaches39. However, many tutoring programmes have high cost. For example, a prominent tutoring programme yielded 0.19 to 0.31 s.d. learning gains at a cost of US$2,500 per child. In low- and middle-income settings, phone calls could provide similar or larger impacts two orders of magnitude more cheaply, enabling scale-up across diverse settings. The SMS-only treatment, when effective, can be extremely cost-effective, at 41.1 learning-adjusted years of schooling (LAYS) per US$100. However, whereas phone and SMS effects were consistent across all countries, the effects in the SMS-only arm varied across contexts. Given this, it seems plausible that the SMS-only intervention represents a cost-effective option in extreme education emergencies where, for example, calls are not an option or schooling is disrupted for such an extended period of time such that any provision of content is substantially better than the status quo.Second, we use LAYS, a cost-effectiveness measure in education that has been estimated for over 150 impact evaluations in low- and middle-income countries; each LAYS can be interpreted as a high-quality year of schooling gained. We find the programme yields 3.6 LAYS per US$100, ranking in the top 10 out of 150 education interventions reviewed47. This result highlights the potential for phone call tutorials to deliver value to education systems and students in a broad array of contexts, both during and potentially even outside education emergencies. Even before the COVID-19 pandemic, there was high demand for cost-effective education programmes to help address the global learning crisis. Our results suggest phone-based targeted tutoring programmes, such as the one tested in this study, have the potential to deliver cost-effective learning gains across contexts and with governments.We also contribute new evidence on the robustness of a cost-effective approach to measure learning—high-frequency phone call assessments. We leverage data across five countries and a series of randomized validity tests. Extended Data Table 5 shows the results of five checks we conducted, described in more depth in the Methods section. Results suggest that measurement of learning by phone can be highly robust, in addition to being cost-effective.ConclusionHere we present results from large-scale randomized trials evaluating the provision of education in emergency programmes across five countries: India, Kenya, Nepal, the Philippines and Uganda. We test multiple scalable models, including government delivery, of remote instruction for primary school children during COVID-19, which disrupted education for more than 1 billion schoolchildren worldwide, as well as during an additional emergency—a typhoon—which further disrupted schooling.Despite heterogeneous contexts, the results show that the effectiveness of phone call tutorials can scale across contexts; we find consistently large and robust effect sizes on learning, with average effects of 0.30–0.35 s.d. In the subset of trials in which we randomized whether the intervention was provided by NGO instructors or government teachers, we find large effects for both delivery models, indicating effectiveness when delivered within government systems (Supplementary Fig. A1).These results have relevance to global efforts to support education in emergencies including conflict, disease, natural disasters and climate shocks, which routinely shut schools down. During these shocks, alternative models are needed to deliver education. The results presented here identify approaches that can scale effectively during disruptions across contexts and cost-effectively improve learning for students. This study further demonstrates that rigorous testing of programmes in humanitarian settings is possible.Even outside education emergencies, millions of children worldwide learn very little in school, either because they are taught curricula beyond their learning level, or because they are unable to access quality instruction since they live in remote areas. Given widespread mobile phone ownership rates globally, phone-based tutoring programmes such as the one studied here have the potential to maintain schooling continuity and accelerate learning even outside emergency settings or after the initial emergency has eased. The low cost, high access and ease of implementation of mobile phone tutoring could build more resilience into education systems, enabling systems to better withstand frequent shocks, and to more generally utilize cost-effective approaches to address a persistent global learning crisis. One high profile example includes Karnataka, India, which is scaling this approach—now called ConnectEd globally and referred to in Karnataka as Ganitha Ganaka—during school and to prepare for school disruptions due to monsoon seasons. Another example includes follow-on trials that built on this study in Latin America54.This study has substantial policy implications. Multiple efforts to scale the intervention studied in this paper are underway worldwide. This paper provides critical evidence to inform effectiveness across settings and delivery models. Future research could examine effects at even larger scale. As Extended Data Fig. 5 shows, the trials covered substantial portions of the geographies studied. However, there is still value in exploring effects at national scale in future work. Future research could examine refinements to the delivery model to engage parents further55 and could evaluate potential spillovers on teacher instruction and student learning in the classroom. Moreover, additional work could explore long-term benefits following observed gains during the study period, with learning potentially compounding over time as skills beget more skills.MethodsDataIn each country, we have two waves of data: baseline and endline. In Nepal, we conducted a baseline survey with a random 50% of the sample, and in Kenya, we relied on administrative data instead of baseline survey data. The endline surveys took approximately 30 min to administer and included approximately 20 questions. These questions included a learning assessment, child wellbeing, parental engagement in educational activities, and parental perceptions of their child’s learning. A portion of the survey was conducted with the parent, and learning outcomes were collected by directly assessing the child over the phone. Endline surveys were conducted a few months after the programme ended. A set of common core questions in the baseline and endline surveys are included in the Supplementary Information.The learning assessment was adapted from the ASER (Annual Status of Education Report) test, which has been used frequently in the literature to measure learning outcomes14,15 and is used routinely across 14 countries. The test consists of multiple numeracy items, including two-digit addition, subtraction, multiplication and division problems. In addition, we asked students to solve a word problem and a fraction problem to capture learning outcomes beyond a core set of mathematical operations. Results are presented in several main results figures and tables. Distributions of raw baseline and endline learning levels are shown in Supplementary Fig. A3.To maximize the reliability of the phone-based assessment, we introduced a series of quality-assurance measures. To minimize family members in the household assisting the child, students had a time cap of 2 minutes per question and we asked each child to explain their work. We only marked a problem correct if the child correctly explained how they solved the problem and enumerators were confident parents were not assisting their child. We also conducted a battery of validity checks to ensure the reliability of the learning outcomes. These results contribute new evidence on the robustness of remote learning assessment data across five countries. Phone assessment has emerged as a common strategy for large-scale household surveys such as the World Bank Living Standards Measurement Survey (LSMS). A growing literature has started to explore the validity of phone-based assessments to measure learning outcomes20,56,57,58, with emerging evidence suggesting phone assessment can capture meaningful information at high frequency and low cost.Extended Data Table 5 shows the results of five checks we conducted on the validity of our main learning outcomes of learning assessments via phone. Column 1 shows the first robustness check where we compare in-person to phone-based assessment for a representative sample of the same students in Kenya. An additional test included back-checks, with a random subset of students tested twice on the same competencies. We find a strong relationship as expected, with large positive coefficients and t-statistics ranging from 5 to more than 20. We further randomize students to receive different problems of the same proficiency (for example, four different questions to measure two-digit addition with carryover). Finally, column 5 shows results from a real-effort question to disentangle effects of the intervention on effort on the test versus cognitive skills. Students were asked to answer several effort tasks, for example figuring out the day of the week or counting zeros and ones. Altogether, results suggest measuring learning by phone can be robust, similar to findings from the World Bank LSMS for household measures of consumption.The survey also included questions on caregiver engagement in their child’s education and beliefs. We measured engagement by asking caregivers how often they spent helping their child with their schoolwork over the previous weeks. We also included a measure of a caregiver’s confidence in their child having made progress in learning over the previous months, and their perception of their child’s numeracy level. Additional questions included information on whether the caregiver has returned to work. We also asked about parents’ demand for remote learning services in the future, and whether they would be willing to pay for such a programme. For students, we asked about child’s mental wellbeing, how much they enjoy school, and the child’s own belief about what mathematics problems they will be able to answer. We also measured non-cognitive skills, such as perseverance and ambition. Results are shown in Supplementary Tables A2 and A3. Finally, we included demographic questions, recording the child’s age, grade and gender.The overall sample size, pooling all sites, is 16,936 households. The flow of participants from enrolment through randomization to endline follow-up is summarized in a CONSORT diagram (Supplementary Fig. A5). For endline surveys, we randomly sampled households from the full sample to interview for a total sample of 12,331. This was due to time and cost constraints. In Supplementary Table A8, we show that those randomly selected for endline interview are statistically equivalent to the full sample at baseline along a series of indicators such as gender, student grade and baseline learning level.Supplementary Table A9 presents the response rate to the endline surveys and an analysis of survey attrition for those randomly selected to be part of the endline sample. The follow-up rate was around 76% of respondents at endline. Supplementary Table A9 also presents a test of whether response rates differed by treatment assignment. This provides evidence that our sample has a high and unbiased response rate.Finally, we also include a survey for teachers to assess their beliefs and instructional practices. These questions include their desire to be a teacher, and teachers’ view that the phone call tutorial programme was helpful for student learning. Questions also include instructional practices, such as involving parents in education further and better targeting feedback to students’ actual learning level. These questions can capture potentially persistent effects on educational systems through teachers changing their beliefs and behaviours beyond the lifecycle of the programme.Experimental designWe ran randomized controlled trials across five countries. Every trial had a control group and a combined phone call and SMS treatment arm. All studies except the trial in India also included an SMS-only treatment arm. In two countries, Nepal and the Philippines, we further randomized delivery of the phone call by NGO instructors or government teachers to assess scalability within government systems. We exploit random assignment to identify causal effects and estimate the impact of these education in emergencies interventions and various scalable delivery models.We first estimate intent-to-treat effects per country c as follows:$${Y}_{i,j,c}=\alpha +{\beta }_{1}{\mathrm{PhoneCalls}}_{j,c}+{\beta }_{2}{\mathrm{SMS}}_{j,c}+\gamma {X}_{j,c}+{\delta }_{s}+{\varepsilon }_{i,j,c}$$