With ever-increasing amounts of complex data, we need compelling ways to distill this information into meaningful, memorable and engaging insights. Data storytelling is an emerging visualization paradigm that aims to “tell a story” with data in order to elicit deeper reflections in an effective manner. However, the effects of adding a narrative to a visualization on the memorability of the information remain speculative. Based on a review of related work, we synthesize a framework of data storytelling principles with concrete actions for every principle. We use this framework to design an online, controlled experiment to test compare traditional data visualizations with data storytelling visualizations in terms of their effects on short-term and long-term recall of information displayed in the visualizations. In general, despite long-held assumptions in the visualization community, we find no significant differences in recall between traditional visualizations and data storytelling visualization. However, we find indications that the cognitive load induced by different chart types and self-assessed prior knowledge on the chart topics could possibly have a moderating effect on information recall.