Overview
What is this tool?
ETO's Research Almanac provides high-level data on trends in English-language emerging technology research, including overall research output, growth, and trends among countries, research institutions, and companies active in R&D. The initial version of the Almanac focuses on topics and applications in artificial intelligence.
What can I use it for?
Use the Almanac to:
- Track trends in research across different fields and subfields.
- Understand how different countries contribute to research in each field and how their contributions are changing over time.
- Explore the impact of research on patents and the broader scientific literature.
- View the most active research institutions and companies conducting research in different fields.
- Analyze trends within highly cited research only, and see how they compare to trends among all research in a given emerging tech field.
What are its most important limitations?
- Our methods for linking articles to topics involve judgment calls. The data on each topic in the Almanac is based on the articles we determine are relevant to that topic using different classification methods, including machine learning models and regular expression-based queries (depending on the topic - see the Merged Academic Corpus documentation for details. These methods involve some judgment calls. We try to apply intuitive, tested criteria when making these calls, but others might disagree with us about what counts as "AI safety research" or "robotics research" (for example). In addition, we rely on statistical models to apply the topic tags. For both reasons, the numbers in the Almanac are necessarily imprecise and should be interpreted as estimates.
- The Almanac only covers English-language research. Other than English, Chinese is the most prevalent language in academic publishing. Unfortunately, due to Chinese policies, it's become difficult for us to access Chinese-language publication data. Because of this, the Almanac’s data source, ETO’s Merged Academic Corpus, contains relatively few Chinese-language publications. Because of this, we decided to limit the Almanac to English-language articles, defined as articles with English titles or abstracts. This excludes a significant amount of global research from the Almanac, and means that Almanac facts and figures related to China should be interpreted especially carefully.
- The Almanac doesn't cover non-public research. The Almanac only covers research that has been publicly released and is included in the Merged Academic Corpus. We believe the MAC covers the large majority of publicly released research worldwide other than research in Chinese, but a great deal of research is never made public. In particular, military research and commercial research may never be written up in open sources, or even written up at all. We have no way of knowing how much of this "hidden" research exists, or how the facts and figures in the Almanac would change if the "hidden" research were somehow included.
- The Almanac's data sources may introduce errors. ETO's Merged Academic Corpus incorporates articles from many different datasets, making it a uniquely comprehensive dataset on worldwide research. No dataset is perfect, though. Errors, gaps, and judgment calls in the MAC's underlying datasets are likely to make it into the MAC itself, and the MAC's merging and enrichment processes may introduce errors. Any such errors affect the numbers in the Almanac. Read more >>
- Research is only one indicator of emerging technology activity. The Almanac focuses on this single, incomplete indicator, so it shouldn't be used on its own to draw general conclusions about which countries or organizations are "ahead" or "behind" in different emerging technology areas. Read more >>
What are its sources?
The Almanac relies on ETO's Merged Academic Corpus. Read more >>
Does it contain sensitive information, such as personally identifiable information?
No.
What are the terms of use?
The Almanac tool and metrics are subject to ETO's general terms of use. If you use the tool, please cite us. The underlying datasets are not publicly available due to licensing restrictions.
How do I cite it?
If you use data from the Almanac in your work, please cite the "Emerging Technology Observatory Research Almanac" and include the link to the tool.
Using the Almanac
How do I use it?
These instructions focus on the desktop version of the tool. Some features may be missing or act differently on mobile devices.
The Almanac is a simple dashboard-style tool. Start by choosing a research topic:
Each topic page has several sections. Scroll down to browse through them, or click a link in the left-hand navigation menu to jump to a specific section.
When viewing line charts, click on an item in the chart legend to hide (or redisplay) the corresponding line.
The "top-cited" toggle
Some sections have a "top-cited" toggle. Click the toggle to display data on the 10% of articles in each year with the most citations, rather than all articles. (Articles are assigned to years based on their date of publication.)
Coming back to a view
When you click on a link in the left-hand navigation menu, your browser's address bar will update to reflect the section you're viewing. Copy the URL in order to return to the same view later:
What can I use it for?
Track trends in research across different emerging technology fields and subfields.
Understand how different countries contribute to English-language research in each field and how their contributions are changing over time.
Explore the impact of research on patents and the broader scientific literature.
View the top research institutions and companies conducting English-language research in different fields.
Analyze trends within highly cited research only, and see how they compare to trends among all research in a given emerging tech field.
What uses are not recommended?
Analyzing specific subtopics, research institutions, or companies in detail. The Almanac provides high-level information on a limited number of emerging tech fields and actors. For more detailed analysis, we recommend using the Map of Science or Country Activity Tracker.
Determining which countries or organizations are "ahead" or "behind" overall in emerging tech. Emerging technology activity is multifaceted. Public research output is one of these facets, but there are many others, from commercializing and deploying new technologies to building institutions that can effectively use and govern them. Understanding how countries, research institutions, and companies compare and compete and in emerging tech requires understanding all of these facets. Because the Almanac focuses on a single indicator, it shouldn't be used on its own to draw conclusions about the overall state of play.
Drawing definitive conclusions about China's overall research output. The Almanac only covers English-language articles (defined as articles with English titles or abstracts), but Chinese authors and organizations often publish in Chinese. A full accounting of China's research output would need to consider these Chinese-language articles as well as English-language articles.
Sources and methodology
The Almanac relies on article-level metadata from ETO's Merged Academic Corpus (MAC). When you visit a topic's Almanac page, you'll see facts and figures derived from English-language articles from a recent five-year period that are tagged with that topic. To learn more about how we apply topic tags to MAC articles, visit the MAC documentation.
It's important to note that emerging research topics have fuzzy boundaries; there's no objectively correct answer to whether a particular article is "AI safety" research (for example). For each emerging topic, we try to capture articles in the MAC that subject matter experts would consider highly relevant to the topic in question. This inevitably involves some judgment calls. In addition, we rely on statistical models to apply the topic tags. For both reasons, the numbers in the Almanac are necessarily imprecise and should be interpreted as estimates. Read more >>
For purposes of the Almanac, we isolate English-language articles by filtering out every MAC article without an English-language title or abstract.
Maintenance
How is it updated?
The Almanac updates as the Merged Academic Corpus is updated. Read more >>
How can I report an issue?
Use our general issue reporting form.
Credits
- Concept: Zach Arnold, Jennifer Melot, Dewey Murdick
- Design and analysis: Zach Arnold, Brian Love, Jennifer Melot
- Engineering: Brian Love, Jennifer Melot
- Review and testing: Catherine Aiken, Liam Alexander, Zach Arnold, Shaun Ee, Brian Love, Jennifer Melot, Neha Singh, and anonymous testers
- Documentation: Zach Arnold
- Maintenance: Brian Love, Jennifer Melot
Major change log
| 11/22/23 | Major data update following MAC changes |
| 5/19/23 | Initial release |
Sign up for our newsletter to receive the latest news and announcements.