Labscape: Design of a Smart
Environment
Larry Arnstein, University of
Washington
Gaetano Borriello, University of
Washington and Intel Research laboratories at Seattle
February, 2002
http://labscape.cs.washington.edu
Introduction
Labscape,
a project at the intersection of bioinformatics and pervasive computing
research, represents a new point in the design space of smart environments.
From the bioinformatics perspective, the goal of Labscape is to make laboratory
work easier, while enhancing the ability of biologists to communicate and
collaborate with each other. From a ubiquitous computing perspective, this
project has provided the opportunity to study the design strategies and
tradeoffs that are associated with building smart environments for real users.
Many of the smart environments built by the
ubiquitous computing research community serve primarily as platforms for
technology evaluation [1][2][3][4]. These environments are crucial to the research
enterprise because they provide a sandbox in which experimental technologies
can be safely tested and evaluated. In contrast, smart environments that are
built to meet the rigorous requirements of an authentic user community provide
insights into effective design and integration strategies that lead to usable,
extensible environments. These insights in turn guide and influence
technology-based research. We present Labscape, now going into a second
deployment, as a case study of smart environment design.
1. The
Application Domain
A cell biology experiment involves observing how the
state of a cell changes in response to some form of stimulus or treatment. The
outcome of an experiment usually takes the form of charts or images associated
with measurements that corresponds to features of the state of the cell. As an
example, the image of Figure 1 might indicate the affect on gene expression
(RNA production) that ten different drug candidates had on otherwise identical
cells. Columns correspond to the cells treated by different drug candidates,
and the rows correspond to gene activity. The darkness of the band at each row-column
intersection indicates the activity level of a specific gene under the
influence of the drug candidate. The image is produced by a technique called
gel electrophoresis, in which an electric field is used to sort molecules by
size. Larger molecules move more slowly through the gel than smaller ones. Since each gene produces RNA molecules of
different sizes this technique can be used to discriminate between them.
Figure
1. An experimental result: an electrophoresis gel
In a common biochemical procedure called polymerase
chain reaction (PCR), very small concentrations of genetic material (RNA in
this case) are amplified (repeatedly duplicated) so that the presence of the
molecules can be detected using the electrophoresis technique. Thus, the entire
experiment would consist of exposing cells to drug candidates; destroying the
cells and performing PCR on their molecular components; then applying
electrophoresis and imaging the gel to read out the results. Many details of the experimental procedure
would have to be communicated for a third party to understand the outcome of
this experiment: some examples are the identities of the drug candidates, the
history of the cells, and the PCR chemistry.
Even the exposure setting of the camera might be important for comparing
the results two different experiments.
Our
goal is to support communication and collaboration in cell biology by linking
the outcomes of experiments to captured representations of the procedures. This
goal closely mirrors a general ubiquitous computing problem that also motivated
the Classroom 2000 project[5], which is to
“automate the capture of live experiences and provide flexible and
universal access to those experiences later on”[6]. Labscape shares this and many other distinguishing
characteristics of ubiquitous computing, including a high degree of user
mobility and the need to stay focused on physical and intellectual tasks.
Figure 2
exemplifies the information management challenge that we are addressing in
Labscape: that the lab bench is place where information is both created and
needed; yet it remains a largely computer-free zone. As a result, significant
human effort is required to transform laboratory results into a form that can
be understood and applied by others. Labscape addresses this problem by
transforming the laboratory itself into an assistant that captures and
organizes data as the work is performed.
In the next section, we describe the design strategies that have enabled
us to begin establishing authentic user communities in two separate
institutions: the immunology laboratory of the Cell Systems Initiative
(CSI) [7] which is part of the University of Washington’s
Department of Bioengineering, and the cell purification laboratory of Immunex
Corporation.
Figure
2. The lab bench: a computer-free zone
2. Design Strategy
The first obvious challenge when setting out to build a
smart environment is to decide where to start. Should one install a rich sensor
network and start invisibly capturing and processing the data? Would it be
better to install many flat panel displays to provide ubiquitous access to an
more traditional application interface? Or, is a hybrid solution most
appropriate? Given our goal of capturing experiments without distracting the
biologist, we started with a hybrid system that emphasized sensor networks and
recognition systems, while relying on the application interface for error
correction. This approach failed to lead to a useful system for biologists for
two main reasons:
·
To meet our users’ needs, we must capture a logically
structured representation of the procedure, but we could not identify sensor
technologies that would provide the detail, completeness, and reliability required
without dramatically altering the physical working environment. As an example,
one of the sensing challenges we faced is that it is essential to know over
which cm2 area a pipette tip is hovering when a small volume of
liquid is dispensed.
·
Though error correction for a hypothetical reliable
recognizer might require little user input, error detection would still
required extensive user monitoring. This is not a problem when the user would
normally be monitoring system output anyway, as in a speech or gesture
interface to a graphics editor. In the
laboratory, however, monitoring of a recognition-based capture system would be
a new and difficult task to perform in conjunction with the physical and
intellectual demands of the experiment.
Our decision to focus on the needs of laboratory users
rather than on sensing and recognition technology had a profound impact on the
design of our prototype environment. We
faced a typical engineering challenge: to develop a system that delivers a
high-degree of utility in a rich environment and is still useful even in a
minimally-enhanced environment. We
defined the worst-case scenario to be the complete lack of a sensor network,
assuming only the availability of an application interface with intermittent
network connectivity. Under these conditions, experiment capture would have to
be the outcome of voluntary interaction with an electronic laboratory
assistant that provides other immediate benefits to its users. Specifically,
such a system must satisfy the following requirement and constraints:
1. Requirement:
It should maximize immediate
utility to the user with minimal required interaction, while producing a
valuable record of the experiment. Thus, the goal of experiment capture
remains, but it has become a side effect of achieving other goals.
2. Constraints:
a.
It must be compatible with almost any cell biology
laboratory. This means that a minimal system must rely only on basic computing
equipment and networking infrastructure.
b.
A minimal system must rely only on mature computer
interface modalities for interaction.
c. It
must be available at all times anywhere in the laboratory with minimal effort
from the user.
Success with a minimally-enhanced configuration will
allow us to establish a user base while we increase the utility of the system
through strategic inclusion of experimental technologies (e.g., ubiquitous
sensing and novel interaction modalities) and through deeper integration with
existing laboratory equipment. The key ethnographic findings that will enable
us to meet the requirement under the given constraints are:
·
An accurate, but abstract representation of a laboratory
procedure has value and does not require sensor networks and device
integration.
- There are many ways that the immediate laboratory
experience can be improved through ubiquitous computation and networking.
3.
Abstract Representation for Laboratory Procedures
Anything
that happens in a laboratory environment has a plausible impact on the outcome
of an experiment: ambient room temperature, how long a sample sits on a lab
bench, how far out of calibration a particular tool or instrument was at the
time of use, etc. Thus we were forced to make a trade-off between the
difficulty of obtaining such details and their potential utility. The key to
rapid development and deployment of a useful system was to discover the highest
level of procedural abstraction in the captured record that still provides
significant utility to the biologists.
Despite
the apparent complexity of the physical environment and diversity of laboratory
instruments and tools hinted towards in Figure 2, we have found that the
essence of laboratory procedures can be represented by arrangements of simple
abstract operations into flow graph structures. For instance, any trained biologist can understand the experiment
plan shown in Figure 3, though significantly more detail might be needed for
them to successfully reproduce the result. More of the features of this
interface are described below.
Figure
3: A Sample Flow Graph representation of a PCR procedure shown in the Labscape
interface.
Though
we do not claim to have identified all possible abstract operations, we are
confident that the list will remain short even as diversity of techniques and
devices increases. Thus, it is reasonable to expect universal standards to
emerge over time. Our list of abstract operations currently includes combination,
incubation, dispensing, separation, measurement, storage,
and retrieval. Figure 3 shows
how these abstract operations are made more concrete by specifying the
appropriate parameters. For example, the separation operation indicates that
the sample will be separated by molecular size to validate the presence of 443
base-pair products from the PCR process.
The
actual complexity and diversity of the physical world can be accommodated in
the language in two ways: 1) by annotating the operations in the flow graph
with unstructured audio, video, text, and images; and 2) by defining less
abstract operations that inherit the semantics of one or more of the base set
of abstract operations. Institutions only have to agree on the basic set of
abstract operations to ensure some level of interoperability and mutual
understandability.
If
biologists could produce an SFG representation of each procedure as it is performed
in the laboratory then we would be meeting our basic capture requirement. The
discovery of the useful SFG abstraction provides the right conceptual model for
interaction with the biologist, but we still have not addressed the need to
provide immediate utility that is commensurate with the cost to the user of
interaction with the system.
4. A Ubiquitous Laboratory Assistant
There
are three phases in the typical laboratory workflow that may be more or less
distinct in practice: preparation, execution, and documentation.
The outcome of the preparation phase is a working (paper) document that the
biologist uses for support in the laboratory. The biologist carries the working
document into the laboratory where it is used as a reference, to track progress,
and to record information during execution of the procedure. In the
documentation phase, a formal record of the laboratory work is entered into the
laboratory notebook. The formal record is produced by transcribing information
from such sources as the working document, previous formal entries, and by
cutting and pasting printed results. The amount of detail in the record is
inversely proportional to the amount effort expended. The documentation work
may occur quite some time after the execution phase was completed leading to
omissions and errors. These three phases closely mirror the pre-production,
capture, and integration phases of defined by Abowd, et al in [5], yet there are significant differences in the
interaction models and technological solutions that apply due to the radically
different natures of the application domains. Labscape has been designed to
maximize immediate utility with respect to each of these three phases.
4.1 The Preparation Phase
Even
in a research environment, laboratory work is highly repetitive. In most cases,
the working document consists of the details that change from one instance of a
procedure to another. Though adequate for that individual’s planning and
recording purposes, such a written record is difficult for a third party to
understand, and moreover, it is physically inaccessible. In Labscape, the
biologist always creates a new complete representation of the intended
procedure during the preparation phase. Input is minimized without loss of
flexibility by allowing the biologist to use any previously completed procedure
as a template for the plan. Like in the paper-based process, the user need only
input the changes. But, unlike the paper system, the outcome is a complete
self-contained description of the procedure that is electronically accessible
and universally comprehensible. Thus, we expect our system to be at least
equivalent in terms of interaction overhead while offering a higher level of
utility.
4.2 The Execution Phase
During
execution, the primary requirements of the information support system are A) to
present the biologist with a clear representation of the plan, B) to provide a
means for keeping track of progress, and C) for recording data and observations
that occur during the procedure. We compare Labscape to paper-based systems in
terms of these three basic requirements, and consider some additional
capabilities that only Labscape will offer in each phase.
A.
Plan access and presentation: Paper-based systems have the
following limitations with respect to access: 1) they tend to compete for space
on the lab bench with the materials and tools needed for the experiment; 2)
they are difficult to transport between work areas when the biologist’s hands
are busy with samples and tools; and 3) as a mobile physical object, they pose
contamination risks when transported in and out of sensitive areas—permanent
notebooks may actually be banned from certain laboratories that contain
radioactivity and biohazards. In contrast, 1) touch panel displays mounted
upright, behind a work area, occupy the biologists’ natural field of view
without obscuring their work; 2) by moving data and state instead of physical
objects, Labscape can eliminate the transportation problem, and 3) reduce
contamination risk.
But,
the main advantage of Labscape is that the SFG provides a window into a
database that can answer a variety of questions that might otherwise require an
interruption. Here are two examples of questions raised by the biologists in
the laboratory during trials that have required reference to an information
source that was not immediately available, but which could have been answered
directly by a Labscape database:
“Show
me if this control sample I am using was positive or negative in the original assay”
“Show
me the volume scaling factor that I used last time I had a low DNA
concentration reading on the spectrophotometer.”
For
presentation and access, Labscape offers unique capabilities in terms of both
interaction overhead and utility. Because of the ubiquitous database access,
utility increases with more users, though a single user can realize the full
benefits of the system.
B. Progress Tracking. In the
traditional workflow, notations on paper are used to keep track of progress in
a procedure during breaks and interruptions, or when the next step in the
procedure cannot be deduced by looking at the physical setup. For instance,
when transferring clear liquids from one set of containers to another, it may
be impossible to tell which transfers have been completed and which are still
pending. Depending on the situation, the biologist might like to record
progress at varying levels of granularity.
In Labscape, as the user indicates progress through simple touch
interactions, the plan is visually transformed into a record of the
experiment—finer granularity of interaction results in better temporal
resolution in the record. The user of Labscape is free to make a trade-off
between precision in the experimental record and the number of interactions required.
For simple progress tracking purposes, the ability to record temporal
information through touch interactions counts as an increase in utility with
the same or reduced interaction overhead. The standard, electronic
representation of progress in Labscape may allow scientists to more easily
cooperate with colleagues, laboratory technicians, or students at a finer level
of granularity.
The icons outlined in blue in Figure 3 represent steps that have been completed. Red and green outlines correspond to pending steps that may or may not require additional information, respectively, before they can be marked as completed. The boxes on the display pop up in response to touch, allowing the user to see and edit properties of the abstract operations represented by the icons.
C. Recording Information. Biologists need to record information during the execution of an experiment. Some examples are sample and reagent IDs, instrument readouts, observations, or just the name and directory location of an electronic file produced by a camera or other piece of laboratory equipment.
Labscape
offers two major advantages over the traditional workflow: 1) recorded
information can take the most appropriate form, such as freehand drawings,
text, audio clips, videos, or images; and 2) such information can be attached
to a specific component of the SFG representation providing the context for
later retrieval. Attaching a drawing or a spoken comment to a particular sample
at a specific point in the procedure ensures that the note resurfaces during
data analysis that takes place long after the physical experiment has been
completed. Multi-media annotation does not violate our constraints on the use
of mature modalities because interpretation or recognition is not required.
Figure
3 shows how the SFG representation provides an organizing structure for all of
the data and observations that are produced during the experiment. Measurement
results are linked to specific measurement operations in the SFG, sample
identifying information (tags) coming from scanners or direct user input are
attached to storage and retrieval operations, and unstructured
annotations and sensor data can be attached to any sample at any point in the
procedure. New data items, or URLs pointing to new data files, are queued up on
the left side of the Labscape display according to type so that the user can
link them to operations in the SFG through touch. The result of working in the laboratory is a complete, accurate,
universally understandable representation of the experiment.
Figure
3 shows the state of the UI after a reagent has been scanned, but has not yet
been associated with a store or retrieve operation in the graph.
The user makes the association by touching the “Primer A” icon, and then
touching the “Tag” property button in the pop-up window. The same technique
would be used to link operations to unstructured annotations or measurement
results, but in this case, the data item would be represented by a URL.
4.3 The Documentation Phase
Figure
4 shows a spontaneous laboratory notebook entry that resulted from a
biologist’s first use of Labscape. Without direction from the system
developers, the biologist chose to print out two graphical windows from
Labscape in lieu of a manual entry in her laboratory notebook. The lower screen
shot shows details associated with a batch of samples represented by a single
icon in the top-level flow graph view. The gel image pasted on the page can
also be accessed by the URL associated with the last step in the flow graph.
The small amount of writing that does appear on the page is a result of the
fact that upstream protocols are not yet in the system, necessitating some
manual cross-referencing for sample identification. This result strongly supports our requirement that the system be
compatible with existing information management practices, which in this case
is the paper laboratory notebook that they are required to maintain. It also
supports our hypothesis that the abstract representation is adequate for
understanding and communication of laboratory results.
Figure 4. A Laboratory Notebook Entry
5. The Design of Smart Environments
Based
on our Labscape experience, we present a set of strategies that we think may be
generally applicable to the design of smart environments.
5.1 UI
precedes AI
Having
built our system to deliver adequate utility through a graphical user interface
alone, we now have the opportunity to incrementally augment the
experience through selective addition of sensing and AI technologies. As an
example, we have shown that it is possible to enable a handheld electronic
pipetter (a tool for manipulated small amounts of liquid) to wirelessly
transmit aspiration and dispensation events to a server. One could imagine
flagging the user when a sequence of such actions does not correlate with the
user level task representation, thereby catching a source of error that is
difficult to trace. The use of sensing and recognition technology to catch
exceptions in this manner would add significant value without placing a new burden
on the user.
As a
more generally applicable example, we are deploying an active badge-like [8] system for location tracking to control application
migration and allocate laboratory resources. Until this system is enabled,
users must touch the GUI button on nearby displays that are labeled with their
name. This manual system is adequate but not ideal because it requires user
interaction, and because it cannot provide intermediate movement data that
would also be useful. However, a system that relies only on the active-badge
might be worse because the user would not be able to correct system mistakes.
By running both systems in parallel, explicit user interactions can be used to
train the location system, possibly eliminating the need for extensive
configuration typically associated with such systems. As the error rates fall,
the system can become increasingly proactive without ever eliminating the user
button. In Labscape, we will begin to rely on sensor data and AI techniques to
further improve the utility and reduce the need for interaction in our system.
But, the training data must come from regular authentic use of a existing
system.
5.2
Values matter
HP
Labs’ CoolTown Exploratorium [9] application shares important characteristics with
Labscape: both support mobile individuals engaged in tasks that demand physical
and intellectual involvement. Despite these similarities, the two projects have
different outcomes because of user values. In the Exploratorium, interaction
with the physical exhibit is the point of the experience—a captured record of
the experience has value, but not at the expense of distraction from the
exhibit. In biology research, the value is in the record more than in the
experience. Interaction requirements that contribute to a better experiment
record are acceptable, especially if the laboratory work can be simplified as
result. As a consequence of these divergent values, HP’s interface has become
increasingly implicit [10], and ours has become increasingly explicit. We are in
the process of installing Labscape into a Seattle public high school biology
laboratory in which the value system is more like the Exploratorium than it is
like our research laboratory. In this setting, Labscape will be cast in a
tutorial role to reinforce the skills and techniques associated with the
process. The organizing structure of the SFG will be used to present
multi-media demonstrations or explanations that have been captured by a
teacher.
5.3
Invisible Infrastructure
Though
the visibility of the Labscape interface is high, and will likely remain so, we
have gone to great lengths to ensure that the infrastructure is transparent.
For example, we did not want our users to worry about persistence and lost
work. There should be no need for explicit file I/O or for defensive backups of
the state of the application. To achieve this level of reliability, all user
interactions that change the state of the experiment plan or record become
transactions against an XML database. But, to prevent the fluidity of the UI
from suffering due to database access delays or intermittent network
connectivity, and to allow the application to migrate at will, [kep1]the
database is decoupled from [kep2]the UI
through an asynchronous event interface. This design approach raised a host of
synchronization challenges that we simply could not address using standard
system tools like networked file systems and TCP/IP sockets. Instead we built
Labscape on top of an experimental run-time system for pervasive computing
called one.world [11]. In addition to features widely acknowledge to be
useful for pervasive computing, such as discovery and event based component
interaction, one.world provides novel features that support programming
for dynamic execution environments. This is an example of where we have applied
and evaluated research technologies in our minimal system.
6.
Status and Conclusions
Our
minimal system includes the addition of the following devices to the
environment: wireless active badge (IR) detectors arrayed underneath the first
shelf above the lab bench; active badges worn by the biologists; RFID and
barcode scanners; and pen/touch driven tablet computers (Fujistu Stylistic 3400)
running Microsoft Windows 2000. In addition, the biologists have access to the
system from their desktop computers in their office environment.
We
have completed one round of informal user studies at CSI and are now engaged in
a formal evaluation that will quantify the benefits and costs of using
Labscape. At the same time, we are deploying Labscape into two more sites: the
immune cell purification laboratories of Immunex corporation, and a Seattle
public high school biology laboratory that has PCR equipment. Immunex is
interested in the quality control potential that labscape offers in a procedure
that is crucial to its scientific mission, while the high school is interested
in Labscape for its potential to help students see the big picture of an
experiment without losing the important details.
The reason for our intense focus on users
(biologists, students, lab administrators, etc.) is to create a sustainable
test-bed for evaluating new ubiquitous computing technologies in terms of their
impact on real user experience. What is unique about our model is that we can
evaluate individual technologies through incremental deployment, and assessment
of impact on a well-characterized environment. This model has already been
applied with respect to the systems infrastructure for ease of development and
maintenance, and it will soon be applied to sensing technologies that increase
the precision of the captured record, and to machine learning algorithms that
make the environment increasingly proactive for users and administrators.
Finally, this effort has led to deeper insight into design strategies for smart
environments.
Acknowledgements
The authors would like to acknowledge the support and
contributions from the DARPA Ubiquitous Computing Program, NSF/REC, The Cell
Systems Initiative of the University of Washington, the Intel Research
laboratory at Seattle, Intel Research Council, and Immunex Corporation.
References
[3]
C. Kidd., G. Abowd, C. Atkeson, I. Essa, B. MacIntyre,
E. Mynatt, T. Starner "The Aware Home: A Living Laboratory for Ubiquitous
Computing Research". In the Proceedings of the Second International
Workshop on Cooperative Buildings - CoBuild'99.
[5]
Gregory D. Abowd, Christopher G. Atkeson, Jason
Brotherton, Tommy Enqvist, Paul Gulley, and Johan LeMon. “Investigating the
capture, integration and access problem of ubiquitous computing in an
educational setting.” In Proceedings of the 1998 conference on Human Factors
in Computing Systems --- CHI'98, pages 440--447, May 1998.
[7]
The Cell Systems Initiative:
http://www.csi.washington.edu
[10] Tim
Kindberg, Private Communication (Pending approval).
[11] Robert
Grimm, Janet Davis, Eric Lemar, Adam MacBeth, Steven Swanson, Tom Anderson,
Brian Bershad, Gaetano Borriello, Steven Gribble, and David Wetherall. Programming for pervasive computing environments, Submitted for publication.