Beyond the horizon: immersive developments for animal ecology research

Abstract

More diverse data on animal ecology are now available. This “data deluge” presents challenges for both biologists and computer scientists; however, it also creates opportunities to improve analysis and answer more holistic research questions. We aim to increase awareness of the current opportunity for interdisciplinary research between animal ecology researchers and computer scientists. Immersive analytics (IA) is an emerging research field in which investigations are performed into how immersive technologies, such as large display walls and virtual reality and augmented reality devices, can be used to improve data analysis, outcomes, and communication. These investigations have the potential to reduce the analysis effort and widen the range of questions that can be addressed. We propose that biologists and computer scientists combine their efforts to lay the foundation for IA in animal ecology research. We discuss the potential and the challenges and outline a path toward a structured approach. We imagine that a joint effort would combine the strengths and expertise of both communities, leading to a well-defined research agenda and design space, practical guidelines, robust and reusable software frameworks, reduced analysis effort, and better comparability of results.

Introduction

Rapidly emerging technologies, such as lightweight sensor tags and advanced satellite imagery, provide unprecedented access to large and quickly increasing amounts of data on animal movement and behavior, as well as the corresponding environmental conditions. The sheer volume, scale, and complexity of the data and the associated uncertainty create challenges for analysis and interpretation. These challenges include questions regarding computer-based handling, such as pre-processing, integration, automated analysis and representation, and human interaction with and interpretation of the data. The wealth of data also creates an opportunity by facilitating the investigation of more holistic research questions, considering several aspects of animal behavior simultaneously with environmental conditions. Examples include supporting machine learning [1,2,3,4,5,6], enriching models of behavior with facets that were previously unresolvable [7,8,9], and providing a broader base for the interpretation and detection of patterns or traits on a more fine-grained spatial and temporal level [10, 11]. This opportunity is particularly welcome at a time when rapid environmental changes, owing to climate change and human impact, and corresponding changes in behavioral patterns might require adaptive approaches and a revisitation of established models and views [12,13,14]. Consequently, an increasing number of tools are available that integrate data on animal movement and behavior, as well as information on the movement environment, with analytical methods to create interactive interfaces (Fig. 1).

Fig. 1

Visual analysis in the TeamWise animal movement analysis tool [15], showing a behavior classification visualization along a movement trajectory on top of satellite imagery. The timeline bar at the bottom shows behavior categories along the time axis, and the behavior annotations are embedded in the 3D view. 2D charts show visualizations of additional abstract data

Meanwhile, we have new technologies that can facilitate data analysis. The development of IT technologies, such as virtual reality (VR) and augmented reality (AR) environments, large high-resolution monitor walls and touch surfaces, holographic displays, and interactive 3D visualizations, have the potential to greatly improve the scope and efficiency of animal ecology analysis. Suddenly, we can superimpose data visualizations on maps on the fly using mobile devices [16] (Fig. 2), “fly with the flock” [17] (Fig. 3, right), or recreate virtual environments for the study of animal behavior in a controlled setting [18, 19].

Fig. 2

Superimposed visualization concept example for movement data analysis on a map, e.g., using AR visualization aligned with a tablet (taken from ref. [16])

Fig. 3

Examples of using immersive technology. Left: Molecular representation on a glass-free 3D Looking Glass device, as a 3D-printed physicalization that can be used for interaction and haptic feedback, positioned next to a standard 2D monitor representation; Center: Interaction with a network visualization in VR; Right: VR view of a flock of storks soaring from a stork’s perspective (taken from ref. [20])

In the endeavor to take advantage of the resulting opportunities, biologists and computer scientists face common challenges. Both seek ways to create scalable and robust solutions [21, 22] for faithful and reliable human interpretation [14, 23]. Similarly, both investigate approaches that exploit new technologies for this purpose.

However, how to best exploit these technologies and integrate them into a human-centered approach has not yet been well defined [18, 21, 24,25,26]. A joint community effort across both communities could save resources, provide joint software platforms, and significantly improve the quality and acceptance of the proposed standards and results.

Immersive analytics (IA) [20, 27] is an emerging research field in computer science that investigates the potential for immersive technologies to be used to improve data analysis and communication, highlighting the potential to reduce analysis effort, widen the range of questions that can be tackled, and improve outcomes. We propose that biologists and computer scientists combine their efforts to lay the foundation for IAs in animal ecology research. We discuss the potential challenges and outline a path toward a structured approach.

Joint research should define guidelines and standards, highlight best practices, characterize the design space for solutions, and address important challenges and research questions. In addition, reusable software frameworks that reduce implementation effort and facilitate reproducible analysis workflows are a major aim.

IA for animal ecology

IA

IA aims to create more engaging and immersive experiences and seamless workflows for data analysis applications [28] by exploiting the affordances of devices and immersive environments (IE), such as user movement tracking, stereoscopic 3D (S3D) [29, 30], multimodal interaction [31, 32], and data physicalization [33]. It is ready to facilitate the analysis of the growing amount and complexity of data in animal ecology research and can provide more efficient and powerful animal ecology analysis tools and environments.

However, to this day, IA is mainly concerned with fundamental research questions rather than practical applications. Consequently, overarching research questions for IA [20, 34, 35] are as follows:

• Use of S3D – how can it be best exploited, which representations are best suited, how does it compare to standard 2D, how to overcome challenges of scale, location, perspective, and depth, and how to integrate classical 2D representations [30, 36]?

• Use of multimodal representation – how can sensory channels be further employed to go beyond the capabilities of visual representation, e.g., through the use of sonification, haptics, or data physicalization [33]?

• Interaction with data representations and the user interface for analysis – how to create efficient and intuitive interfaces, e.g., using multi-modal interaction [37, 38] and transitional interfaces [39, 40]?

• Navigation – what are good navigation metaphors to allow the user to traverse large data sets while maintaining orientation and supporting the generation of a mental map, i.e., the internal representation of knowledge concerned with the data and its connection to external representation in the IE [41]?

• Scalability – how to cope with huge data sets, regarding the computational requirements and responsiveness of automated analysis but also aggregation and abstraction for human interpretation?

• Collaboration – how can collaboration for analysis be explicitly supported, e.g., given the large physical immersive spaces provided by IE [42]?

• Analysis and presentation environments and audience – how to tailor an approach for the constraints of an environment, e.g., lab vs in the field, or a group of users, e.g., experts performing exploratory analysis vs decision makers using communication of results?

Nevertheless, IA has been commended for use in a variety of areas [43,44,45,46,47,48,49], (Fig. 3). However, efficient and user-friendly approaches must be tailored to the specific questions and requirements of an application.

The combination of data visualization, multi-modal data representation, multi-modal interaction, integration of analysis methods, and device and environment characteristics constitutes a large design space. Thus, suitable solutions within this space must be designed and evaluated for use in animal ecology research with a focus on specific tasks, users, and data to fully benefit from the capabilities of IE compared to those of classical desktop environments. To this end, proper use has to be made of the differences in environment characteristics, e.g., regarding the physical immersion of the user, S3D, field of view/regard, user movement tracking, gesture recognition, and interaction using hand-held controllers.

A combination of more intuitive interaction with the data, 3D representations for abstract and spatial data [30], and integrated interfaces for automated analysis methods has the potential to greatly improve the analysis cycle, enhancing user experience and efficiency. Possible examples of how this might work for animal ecology research include observing the environment from an animal perspective, walking through a scene of interacting animals while being able to steer an analysis interactively, and combining these scenes with classical abstract data visualizations (Fig. 4). These examples can also be extended by adding support for collaboration, by providing several analysts with the same data representation and allowing shared annotations and analyses, for example. In addition to software frameworks and in cases such as VR HMDs, full software eco-systems, have been developed around immersive hardware technology, significantly reducing the threshold to prototype and develop immersive visualizations as well as integrate automated analysis solutions. Thus, the current situation provides an ideal foundation for investigating the potential of such environments for animal ecology data exploration and analysis.

Fig. 4

Use of AR technology: Further analysis results and annotations, such as those regarding environmental conditions, can be calculated or interactively retrieved from databases. They can be used as a visual overlay or integrated into the scene. Creating such spatially situated visualizations, i.e., visualizations integrated with a real-world referent such as physical spaces or entities, is a core challenge of IA [50, 51]

Challenges from animal ecology

The relationship between animals and their environment is complex, and animal activity area characteristics depend on the location of suitable living conditions, that is, based on the specific features of an area [52]. Meanwhile, animal behavior can also heavily affect the environment in a variety of ways, such as through pollination, grazing, and the arrival of invasive species. Therefore, current research in animal ecology is also concerned with the identification of features and stimuli that inform animal decisions, trigger actions, influence behavior, and facilitate orientation and navigation. Many analyses, such as those of habitat and corridor configuration, foraging quantity and quality, and migration paths, involve the application of environmental feature data.

The important research questions are often interrelated:

• Investigation of movement and movement patterns on different levels of scale [53,54,55] and individual or collective movement [24], such as home-range, territorial behavior, and swarm movement

• Interaction between individuals or groups [56, 57], e.g., in movement, predation, and decision making [58]

• Impact of environmental conditions, e.g., on decision making, social dynamics, or survival [18, 19, 59, 60]

• Differences in species and groups, e.g., based on phenotypic variation [61] or related to evolutionary relation

• Cognitive processes underlying behavioral patterns, e.g., foraging or mate choice [62]

• Prediction and modeling of behavior [13, 63]

The available information on animal behavior and movement is usually collected by sensors, imaging, and subsequent processing of the results. Analysis methods need to exhibit a certain level of robustness toward incomplete and noisy data and be capable of coping with the uncertainty associated with it. Representations of the results must convey the corresponding restrictions and limits of confidence to analysts.

Further, behavior information usually needs to be embedded into the environment in which it occurs to allow proper interpretation [53, 64]. This combination is often challenging because of the sparsity and quality of the data available for both the animals under investigation and the corresponding environment [65]. The data only represent a part of the animal environment, and which type of information extracted from the data best facilitates modeling of the interrelations between an animal and its surroundings and what confidence level can be reached need to be investigated [66,67,68]. For the design of an analytical concept, methods for the extraction and visual representation of the necessary information need to be conceived. Thus, to analyze animal behavior properly, environmental features must be extracted from the available information, integrated with the automated analysis, and presented in tuitively for interactive exploration by the analyst. Hence, which features can be collected and how they can be provided and integrated into approaches and tools for animal ecology researchers need to be examined.

Given the 3D nature of the animal environment, a representation in S3D is appropriate, and there are indicators that S3D representations have advantages over 2D representations for a variety of tasks [29, 30, 69]. However, such a representation comes with a number of caveats. While the investigation can benefit from the depiction of the natural environment, for example, to create hypotheses for landmarks used for decision-making, the necessary fidelity of the representation is a parameter for IE design. How can the environment be reconstructed for human analysis, particularly when no first-hand experience of local conditions is available? To this end, the integration of imaging methods and subsequent processing for the identification of features is required, and models must be employed to represent and simulate the environment and its features in an IE. Furthermore, whereas interaction with representations in 3D can be designed in a more intuitive manner than in a desktop setup [32], adding interfaces for analysis methods in this setting is a challenge. This includes interfaces for settings and selection, as well as for a well-interpretable representation of intermediate results.

Whereas big data approaches can foster the understanding of the ecology of animal movement and behavior [57, 70], they also provide use cases with specific challenges and requirements for analysis approaches. These include the ability to monitor and analyze large-scale data and to derive patterns and unusual behavior at different levels of scale. One example of such a challenge is the recently introduced concept of “Internet on Animals,” which proposes fine-scale biologging through the combination of WiFi and multi-sensor devices. The proposed architecture supports big data biologging, particularly the collection of movement and locomotion data, over extended periods of time. Thus, the analysis environments for such data must be able to handle the scale of the incoming data. Applications might require real-time monitoring as well as aggregation and abstraction of long-term data for trend and pattern detection, along with comparison, for example, between animals or time periods. Important features of the data required for the analysis must be preserved in the aggregation process and made available to the analyst, either in an overview or on demand [70, 71]. The ability to integrate different levels of scale, for example, from different data collection technologies and sampling strategies [70, 72], poses another challenge. This requires proper handling, representation, and navigation techniques. An example would be smooth transitions between different levels of temporal or spatial scales. Further challenges include surveillance for the monitoring and prediction of environmental changes or events. Interesting use cases include the monitoring of the impact of human land use and climate change and the attempt to predict natural disasters by detecting unusual animal behavior [73].

Potential for synergies and research

IEs have applications in animal ecology education [38], outreach, decision making, and the investigation of behavior. In a holistic approach, the enriched recreation of the environment and information on animal behavior can be combined with access to analysis methods and pipelines, supporting an immersive interactive experience and analysis. For example, Harel et al. [59] discussed the representation of arboreal animal movements and decision making in VR. They mapped the 2.5D setting of canopy environments into a S3D environment for detailed analysis. As the ability to precisely measure movement and position is improving, more fine-grained options for representation are becoming available, which may support a better understanding of animal decision-making, for example, regarding the trade-off between risk and reward [59]. However, as Harel et al. mention, current technologies used to collect data, such as sensor collars or drones, may still have an impact on the available decision options of the animal and the decisions taken.

IEs can provide further advantages for the analysis of such environments by supporting spatial sound representations of collected or simulated data that foster further insight into environmental characteristics, decision drivers, and variations across taxa [74, 75]. Klein et al. [17] performed a benchmark study to assess the suitability of different IEs for animal movement visualization from both the developer and analyst perspectives. They concluded that suitability is strongly dependent on the specific environment and design in relation to the task. Examples include analysis in the field or the lab, collaborative analysis, and decision-making and engagement of the general public.

The use of IEs is not restricted to the representation and analysis of data but can also be used to create environments for controlled studies. Design guidelines and experimental evidence from IA may help to improve environmental design. Sawyer and Gleeson [76] summarized the use of VR for animal behavior investigations in biological laboratories. Their summary includes neurological studies that involved the placement of animals in a VR environment for controlled behavior studies, as well as studies that promoted the investigation of virtual animal models to replace traditional animal models in biological laboratories. Corresponding results might also inform studies on human-animal interactions, for example, a review on human-dog interactions in VR and AR presented by Oxley et al. [77]. Computer-mediated visual stimulation of animals for behavioral research has been employed for many years, and VR was successfully adopted several years ago. For instance, see the survey on VR systems for rodents presented by Thurley and Ayaz [78]. In this study, they advocate for the transfer of concepts tackling research questions of spatial cognition and navigation from human to animal behavior research. However, Thurley and Ayay also discussed the issue of the trade-off between stimulus control and restraint, which is greater for animals as they cannot be instructed before a study. Restraints, such as movement fixation, might restrict movement options, limit necessary sensory input, or even lead to unintended deviations from real-world experiences, including conflicting sensory information, which might be a confounding factor in the investigation. Taube et al. [79] discussed the problem of using results from virtual setups to interpret spatial orientation and navigation, as these setups do not factor the activation of motor, vestibular, and proprioceptive systems. To overcome these restrictions, Stowers et al. [80] presented a VR system for freely moving animals. Naik et al. [18] presented a review of animal behavior experiments conducted in virtual environments and argued that while virtual environments have become a widely used tool for animal behavior research, more interdisciplinary research is required.

Open research questions in IA for animal ecology include how to integrate data on animals and the environment into automated analysis, which IE is best suited for a specific analysis task, how to best represent data and analysis results within the IE, and how to support the exploration of the data through intuitive interaction and navigation approaches (see Toward a structured approach for animal ecology IA section). Only initial investigations from the computer science field have targeted aspects of animal ecology, particularly regarding geo-visualizations. Examples include the use of globes and maps [81] and differences in IEs for bird movement analysis [17].

Given the opportunities of IEs, including not only the larger visualization space and up to six degrees of freedom for the analyst’s movement in the data representation, but also more intuitive interaction, several advantages for animal ecology are envisioned. First, the 3D visualization space can be used to better show animal behavior in the environment in which it occurred (Fig. 4a). Next, the analyst can be immersed in the scene if necessary, exploring different perspectives, for example, from an animal’s viewpoint. In addition, interaction with the scene can be more direct; an example would be selecting animal representations by hand for further analysis. Finally, the integration of automated analysis into scene depiction makes the switch between result representation, data exploration, and interaction with analysis methods unnecessary.

However, the value of such solutions depends on how they support the complex analysis of animal behavior and the interplay with the environment, how they can be integrated into everyday analysis work, and how they can intuitively be used by domain experts. Analysis aspects, such as interactions between animals, movement patterns, influential environment characteristics, and drivers of animal decisions, must be integrated in the analysis and properly represented for human reasoning. We assume that different use cases will benefit quite differently from affordances of IEs and, thus, will need specific design solutions.

The main categories that distinguish the use cases include the spatio-temporal and environment conditions of the observation and analysis (Fig. 5). The observation can be

• direct, i.e., an analyst observes the animal through the senses and also perceives the context of the surrounding environment, or indirect, in which sensors are used to observe a selected set of certain features and parameters of animal behavior and environment;

• in real-time, for instance, when an observer is in the field simultaneously with the animals under observation, or asynchronously, i.e., with a delay between the animals’ action and the observation of its results or remainders (e.g., tree marking or nest construction);

• in the field or in a controlled environment, for example, with a specific experimental setup in a confined lab space to test a hypothesis.

In particular, for direct observations in real time, the observer might also be the analyst or prepare information for later use by an analyst.

Fig. 5

The aspects that distinguish use-cases for application of IA in animal ecology cover several major categories related to location, time, and environment

The analysis can be

• collocated, i.e., in the same space as the animals, or remote, for example, in the analyst’s office or data theater,

• in real-time, i.e., while observing animals, or asynchronously, for example, by using a replay of the behavior or investigating collected data at a later point in time,

• in a purely physical environment, or computer-mediated environment, for example, fully computer-mediated VR representations of animal behavior in an environment, AR superimposed visualizations, or a hybrid setting, e.g., by projecting an overlay of the environment in which the behavior occurs on an office desk (Fig. 4a),

• in the field or in a controlled environment similar to that of the observation.

To foster the active development of corresponding solutions, a requirement analysis and an exploration of concepts, methods, and designs are required. These steps should be performed in a joint effort by biologists and computer scientists. Thus, research could greatly benefit from focused collaboration between both groups to define a design space for solutions.

To properly assess possible designs for IE, we can first analyze the use case at hand and its requirements regarding the above categories, resulting in a combination of features such as direct and real-time observation in a controlled environment with asynchronous analysis in a VR environment. Afterwards, fitting designs and technology options can be chosen to support the analysis, as well as the observation, for example, by a data overlay through AR in the field.

Some of the resulting possible combinations are more promising than others, and for some, the potential benefits are unclear. For example, the use of VR in a direct observation setting seems artificial because the use of VR hinders direct observation. However, in a collaborative setting with multiple analysts, of which only one uses VR, the case might still be justified. However, in general, we assume that an AR setting is more suitable for direct observation and collocated analysis. Note that mixed scenarios of these categories are also possible, for example, in such a collaborative setting in which roles in a team are distributed.

Another important question for IA research and system development is how to measure success, considering the often complex workflows and long-term cyclic processes that are targeted.

Interdisciplinary collaboration

While there are already ongoing and successful collaborations between computer scientists and animal ecologists, the current state of research offers the opportunity to shape the research direction of IA for animal ecology by laying the foundation for a community effort. In this effort, the communities can work together to identify the main challenges and coordinate work on common standards and platforms, fostering better exchange, reuse, and comparison of approaches. There are large differences in methodology, vocabulary, and approaches between the domains of biology and computer science that need to be bridged [82]. Computer scientists are concerned with research questions regarding the methodology and concepts of computational approaches to create effective and efficient methods for data analysis. A large part of today’s work in animal ecology, particularly regarding data processing and analysis, involves computer technology, and many animal ecologists are also early adopters of new technology [21]. While often proficient in practical programming, they need to use such methods to tackle their research questions and improve reproducibility and replicability of studies [83]. However, they often have to deviate from their research focus on computer science, and large efforts have contributed to ad-hoc solutions that are often not reused. Rather than having ecologists spend a considerable portion of their time learning and re-implementing advanced computer science concepts, we suggest developing a unifying framework that targets typical use cases and provides guidance, and establishing implementations of best practice approaches. Therefore, we advocate a structured approach to lay the foundation for developments that exploit the skills and expertise of both sides. Initial steps in this direction have been taken [21, 55, 65].

Instead of selecting existing standard tools, with all their restrictions, animal ecologists and computer scientists together can develop tools and software tailored toward the specific requirements of animal ecology. Such an interdisciplinary collaboration would also allow large-scale efforts to be targeted, such as a common platform for animal ecology IA, which decreases the implementation effort for biologists, speeds up the development process, leads to better-designed analysis environments and well-characterized analysis workflows, and increases analysis efficiency and reproducibility. Interactive interfaces that support the creation and execution of individual analysis workflows in a programming-free manner could decrease analyst effort and improve analysis design, as shown by MoveApps [84] or the Orange [85] data-mining platform.

Success stories for structured interdisciplinary approaches to data analysis exist in other areas, where large interdisciplinary teams define ontologies and standards and frameworks and software libraries provide well-designed analysis tools. Examples include the COMBINE initiative [86, 87], which coordinates the development of community standards and formats for computational models in systems biology, and the Bioconductor framework [88], which provides tools for the analysis of high-throughput genomic data, showing how coordinated long-term efforts can provide interoperable standards, foster exchange, avoid duplicate work, improve the quality of available software and results, and facilitate high-quality communication of said results.

Toward a structured approach for animal ecology IA

To initiate a larger initiative, a community effort should be undertaken to perform a requirement analysis that characterizes the available and expected data, analysis operations, and workflows in a structured manner. We suggest the formulation of guidelines and creation of frameworks for the collection, storage, and processing of high-quality data to improve the data analysis. Existing concepts from animal ecology research can be extended and improved with the results of computer science research, such as computational modeling, visual representation, and computational efficiency. For example, the variety of pipeline and workflow models can be extended by aspects that have great potential to improve reasoning and decision-making, such as the way analysts interact with the data, encoding used for data representation, and environment within which the analysis is performed.

Conversely, computer science methods should be extended to address semantics in animal ecology. Many concepts for data modeling, analysis, and representation that have been developed in computer science might be used as the starting point for solutions tailored towards animal behavior research, for example, by integrating semantics and classifications. Thus, standard pipelines and workflows (Fig. 6) should be reinvestigated, enriched, and refined to focus on the specifics of animal ecology. The sensemaking loop model for intelligence analysts [89] is a great example of a refined model that considers the concepts and terminology of the application area. Moreover, it models the process in terms of several cyclic processes, in contrast to basic pipeline models, which often miss the cyclic aspect of data analysis. Motivated by such examples, we propose to characterize the specifics of animal ecology to provide a design space for practical solutions tailored toward IA for animal ecology.

Fig. 6

Flow chart of data processing and interactive analysis for animal ecology

Important aspects that should be considered are data pre-processing, provenance, analysis of spatial-temporal data, collaboration, rendering of a 3D environment, human-computer interaction, environment, encoding, notations, and standards for representation, reporting, and exchange [90]. Whereas such aspects are common in computer science, their combination in the context of decision-making and living organisms is unique.

In the following, we discuss the selected aspects that should be tailored for analysis in animal ecology.

• Data integration and pre-processing: The amount and complexity of incoming raw data is a challenge for analysts and established methods [14, 72, 91]. Different types of data such as data on movement, animal physiology, and the surrounding environment are collected as time series, images, videos, scalar fields, and point clouds. Combinations of such data are required for efficient and effective automated and human analysis [92], and well-specified procedures for the handling of missing data, outliers, inconsistencies, and uncertainty need to be performed. To improve both types of analysis, current approaches need to be adjusted with measures to test/support the data quality, such as automated preprocessing, storage requirements in databases, annotations on the applied processing, mapping of data from different sources, aggregation, specified formats, and standards and ontologies to structure the data. While this challenge has already been tackled for standard environments, some IEs exhibit different limits of scalability and technical restrictions regarding the integration of data pipelines. Moreover, when data representations are designed, information on the preprocessing steps might be required for the analyst to avoid misinterpretation, and robust approaches are required that can cope with quality issues in the data [71]. The integration of data from multiple levels of organization can be used to improve the accuracy of the subsequent analysis [72]; however, careful design is needed, not just for the integration, but also for the corresponding user interface.

• Automated analysis: Given the rich set of already available analysis tools, for instance, provided as R packages, python packages, and machine learning-based methods, solutions for IEs should take advantage of those tools, and concepts are required to provide interfaces that allow seamless integration. This includes not only providing computational access but also considering how a user interface needs to be designed to support a smooth workflow, for example, allowing parameterization of methods. Furthermore, how should the established ways of representing results, such as traditional 2D charts and plots, be transferred into IEs needs to be investigated. As a side effect, such an integration would ease the load when analysts switch between different environments, for instance, in transitional user interfaces [40] where different types of mixed-reality are available to the analyst.

• Cyclic analysis workflows and provenance: The analysis process might be based on an already pre-processed and well-defined set of data that is used for a series of different research questions. For each of these questions, a researcher might revisit the data to explore different aspects. Thus, solutions that make the history of previous investigations available and allow the storage of annotations and partial results are required. In practice, often a cyclic approach is taken [89, 93], in which previous analysis results and additional data can be fed into the investigation. Knowledge generation models focused on animal ecology might help shape workflow design.

• Visual data representation: Available data and technologies facilitate new visual metaphors that might improve the quality and efficiency of the data analysis [94]. However, the potential for animal ecology has not yet been well investigated. For example, whereas publications advocate for the representation of animal movement in the context of the environment in which it takes place [17, 90, 91], the actual impact and optimal ways to do this are not yet clear. For instance, does 3D help or rather distract from the task at hand? Is environment imagery helpful if it does not reflect the situation at the exact observation timepoint? The use of graphical notations to facilitate interpretation and improved comparability is established in many research areas, such as software engineering (unified modeling language) or systems biology (systems biology graphical notation) [94]. Similar efforts could be undertaken in animal ecology to improve analysis and communication and support replicable and well-defined results. With more data sources and dimensions available, proper use of visual variables for data mapping in different IEs, as well as the extension to multi-modal representation, for example, the inclusion of haptics or sound, should be investigated. The visualization of data quality, regarding missing data or uncertainty in measurements, is a further aspect that needs to be considered. When data representations are optimized for IEs, for instance, the use of S3D that allows the user to walk in the representation, the questions of how to share such representations and how to communicate the findings arise. While established charts could simply be printed or shared as images, this might differ greatly for 3D representations.

• IEs: As we have discussed in IA for animal ecology section, the selection of fitting IE designs would strongly depend on data characteristics, research and analysis workflow, and the specific research question under investigation. Which technologies to use and how to employ them needs to be investigated for specific settings in animal ecology. For example, the different classes of animals, available data sources, nature of the investigation, e.g., exploration or hypothesis testing, and the specific task, e.g., investigation of use of resources or group interaction, should be considered. Consequently, we would expect guidelines on what design is better suited for which type of analysis, going beyond the few results for specific classes of animals and environments [17, 81].

• Interaction: We believe that new technologies allow the creation of more engaging interfaces but also require further efforts to design interaction metaphors that support more intuitive, effective, and efficient data analysis and reasoning by animal ecologists. One example is to show the data in a representation of its environment, for instance, in a table-top or room-sized representation, with navigation by movement or gesture, or adaptive representations, facilitating the manipulation of data representations and interface elements in a more natural way. However, the efficiency of interactions and the required amount of user guidance needs to be investigated.

• Collaboration: Collaboration should be specifically integrated in practical approach for animal ecology [95]. This includes data annotation (classifications, personal comments), recording and reproduction of processing steps (provenance) for synchronous and asynchronous collaboration, and methods to support collaborative analysis in real or virtual spaces, for example, through the representation of collaborators to facilitate communication and exchange.

Action items

The following steps foster a better mutual understanding and collaboration between the animal and computer sciences. While they constitute research in their own right, they can serve as a preparation for further research on both foundational and practical solutions for animal ecology research.

To exploit the potential of new immersive developments in animal ecology research and lay the foundation for unified handling and reproducible results we propose the following:

• Conduct a requirement analysis that covers a broad range of use cases, discussing characteristics of data, tasks, workflows, user roles, research questions, and potential analysis environments. The different aspects of such an analysis are summarized in Fig. 7.

• Evolve existing guidelines and pipelines for data processing and interactive representation or create new ones that fit better the requirements of practitioners in animal ecology research regarding the above listed characteristics.

• Have a close collaboration of biologists and computer scientists to drive the exploration of the design space in the right direction—developments have to be informed by the experience and needs of the practitioners. Thus, common venues should be organized to foster exchange and plan the path ahead.

• Create software support by developing a unified framework that allows for easier implementations, better reproducibility, and comparison of results, as well as a unified user experience. Design considerations based on the requirement analysis can inform a framework architecture in line with analysis workflows.

• Motivate the development and use of notations and standards for storage, exchange, automated analysis, and visual representation.

Fig. 7

Main aspects of the requirement analysis, including dependencies on data, user, task, and environment, as well as existing approaches and workflows of practitioners

Conclusions

We see large potential for the use of IA approaches for animal ecology research as well as in the application of concepts such as standardized notations. However, we think that possible avenues are underexplored and research could greatly profit from a structured collaboration between the animal ecology and computer science fields in the topic of IA. This would allow the enrichment of models and concepts from computer science with the requirements of animal behavior research and shape reusable and durable solutions, such as a unified framework, to exploit the new immersive technologies for data analysis. The first steps could include workshops to foster the exchange and define the challenges and a roadmap, followed by specifications of guidelines and standards, for which interfaces and software platforms can be created.