Table of Contents
This is work in progress; this document has not yet been reviewed and approved. See To Do section.
The Meta-Environment is a powerful system that can be viewed from different angles. From the angle of the end-user who creates languages and tools using the graphical user interface, from the angle of the power-user who builds applications using the command line interface and the system libraries to the angle of the system developer who is maintaining and extending the system. For all these stakeholders we present architectural views that will help them to better understand the system and its possibilities.
The Meta-Environment is a framework for language engineering (language design and implementation, source code analysis and source code transformation) consisting of:
Syntax analysis tools.
Semantic analysis and transformation tools.
An interactive development environment.
The Meta-Environment is an open framework that
can be easily extended with third-party components;
can be easily tailored, modified, or extended;
is supported by an open source community.
The Meta-Environment is a generalization of the ASF+SDF Meta-Environment that has been successfully used in a wide variety of analysis, transformation and renovation projects.
The Meta-Environment can be used for many different purposes, including:
Parsing (new and old) programming languages, for further processing the trees, (e.g. COBOL, C, Java, PL/I, SQL)
Analysis of source code (fact extraction, type analysis).
Transformation of source code.
Generation of source code.
Design and implementation of domain-specific languages.
Generation and rapid prototyping of IDE's (Integrated Development Environments) for programming languages and domain specific languages
Generation of documentation from source code.
Compilation of domain specific languages (DSLs).
Formal description of the syntax and semantics of (programming) languages.
The objectives of The Meta-Environment can now be summarized as follows:
To give non-expert users access to the above mentioned techniques and application areas.
To provide expert users with a powerful tool suite.
To create a flexible testbed for experimentation with new research approaches and technologies.
We identify the following stakeholders of The Meta-Environment:
End-users that only use the system via the graphical user-interface (GUI).
Power-users that use both the GUI, the command-line interface of the system as well as libraries and Application Programming Interfaces (APIs) provided by the system.
System developers involved in the design, implementation, testing, maintenance and deployment of the system.
The following views on the system will be described:
The end-user view (USR) describes the system as an interactive black box.
The decomposition view (DCP) describes how the system is organized in components.
The coordination view (CRD) describes the cooperation between components.
The detailed coordination view (DCRD) describes the high level design of several main coordination processes within The Meta-Environment
The data layer view (DAT) describes shows the relation between datatypes.
The package view (PCK) describes the mapping between architecture elements and implementation packages.
The package dependency view (DEP) shows how the various packages use each other.
The power-user view (PWR) describes the system as a suite of commands and libraries.
The PIMP Meta-Environment view (PMV) describes the difference of the MetaStudio based Meta-Environment and Eclipse-IMP based Meta-Environment.
Obviously, not all views are relevant for all stakeholders. Table Mapping between stakeholders and views shows the details.
Key: d = detailed information, s = some details, overview information, x = anything
The language elements are formalisms for specification, programming or scripting. Programs in these languages are read and written by humans and are intended to solve specific problems.
ASF. Algebraic Specification Formalism. This a notation for describing rewrite rules and is mostly used for defining software analysis, fact extraction, and software transformation.
ASF+SDF. The combination of the formalisms ASF and SDF. ASF+SDF can describe both the syntax of a language and the operations on that language (checking, execution, analysis, transformation).
the syntax of the base language,
the functions that can be applied to base language programs.
Note that the ASF+SDF modules may define several base languages at the same time.
the syntax of the base language (e.g., C, Java, Boolean expressions, a domain-specific language),
the functions that can be applied to base language programs (e.g., typecheck, extract facts, compile, remove unused methods).
The input term can be freely edited and is checked for syntactic correctness before any function is applied to it. It is possible to simultaneously edit different input terms in different base languages.
Output term. A text that describes the result of applying a function to a program in the base language. Note that this text conforms to the syntax R, where R is the result sort of the function that was applied. With Java-to-Java transformation the result sort will be Java. When no function is applied, the output term is identical to the input term.
Rscript. A small scripting language for defining relational expressions. Used for the analysis of facts extracted from software.
Rscript is not yet integrated in V2.0.
SDF. Syntax Definition Formalism. A notation for describing the grammar of programming and application languages.
Tscript. The script that describes the cooperation between components in a ToolBus-based application.
The data elements are formats for representing domain-specific data. The corresponding data are written and read by programs.
The AsFix format is a full parse trees that contains all the original layout and comments from the original source code program that was parsed.
The AsFix format is self-descriptive: each subtree contains information about the exact grammar production that has been used to parse the text that has resulted in that parse tree.
The AsFix format does not contain source code coordinates per se, but a separate tool (addPosInfo) can easily compute these coordinates and add them to the parse tree in the form of annotations.
ATerms are language-independent and can be processed by programs in any language.
ATerms can be annotated with auxiliary information that does not affect the tree structure.
ATerms preserve maximal subterm sharing. This means that common parts of the data are not duplicated but shared. This leads to considerable size-reduction of the data.
Box. Intermediate representation of the prettyprinter. A parse tree is first converted to a box term that includes all desired formatting directives (alignment, font and color directives, and the like). Next, the box term is converted to various output formats (plain text, HTML, etc.).
Compiled specification. It is possible to compile the ASF+SDF into a very efficient executable form. Compiled specifications can be used via the command line interface of The Meta-Environment by power users.
The processing elements are "active" parts of the system that usually transform inputs to outputs. Inputs and outputs may both be machine or human readable or writable.
APIGEN. Application Programming Interface (API) generator. Given a datastructure definition (in the form of an Abstract Data Type), APIGEN generates C or Java code to access that datastructure. Internally, the datastructure is represented as ATerm.
ASF+SDF checker. The ASF+SDF checker checks an SDF definition for compatibility for use with ASF, and also checks some production attributes that are specifically interpreted by ASF rewriting engines. It is an extension of the SDF checker.
Configuration manager. The configuration manager handles user settings and preferences.
Graphical user interface (GUI). The graphical user-interface (GUI) gives end-users access to the system's functionality. It is a "sovereign" user-interface that occupies the complete desktop window and provides functionality like:
Opening, editing and closing ASF+SDF modules.
Opening, editing, reducing and closing input terms.
A graphical and tree-structured display of the import relations between ASF+SDF modules.
A graphical display of parse trees.
Error and progress indications.
Parser. The parser takes a parse table (as produced by the parse table generator) and text (as provided by a text editor) as input and produces a parse tree as output. Any errors are shown in the error display of the GUI.
Prettyprinter. The prettyprinter converts parse trees to text. The prettyprinter uses default rules to insert layout in a parse tree so that its corresponding text is presented in a uniform way. Optionally, the ASF+SDF specification may contain formatting rules that can replace this default behaviour.
The prettyprinter is not yet fully integrated in V2.0.
Sisyphus. A system for continuous integration that rebuilds the system after each change that is committed by a developer.
Structure editor. A syntax-directed editor that closely cooperates with the text editor. It is mostly used for syntax-directed navigation through the text. The structure-editor does not appear as such in the GUI but all its functionality is visible through the text editor.
Term store. The term store contains all parse tables, parse trees and other intermediate data that is generated during execution of The Meta-Environment. This includes:
The parse tree for each ASF+SDF module.
The parse tree for each parsed input term.
The parse tree for each generated output term.
The Meta-Environment. The architecture of The Meta-Environment (or just "the system") is the primary object of study of this document.
ToolBus. The ToolBus coordination architecture enables the flexible and controlled combination and orchestration of software components. It is used as backbone for The Meta-Environment. The ToolBus has the following characteristics:
Components (or tools in ToolBus parlance) can be written in different programming languages.
Components can be running on different machines.
All interactions between components are regulated by a ToolBus script (or Tscript for short) that is executing in the ToolBus. Tscript is a concurrent language that allows the definition of parallel processes, messaging between these processes and interaction between processes and tools.
The following quality attributes guide the design of the architecture of the Meta-Environment.
Usability. We want to support our users at all levels of experience in as many ways as possible:
Through a GUI that automates many of the tasks involved in language engineering. For instance, the user only sees a grammar and the fact that it can be used for parsing but is unaware of the fact that parse table generation is going on behind the scenes. The same is true for rewriting, formatting and the like.
Through a command line interface that allows the batch-like combination of components.
Through libraries that package specific functionality that can be called directly from user-written programs.
Through ToolBus components that can be included in interactive applications.
Adaptability. The Meta-Environment is at the same time a framework for building applications in the area of language engineering and a testbed for doing research. This implies that we want to be able to combine existing components in new and flexible manners and to enable the easy integration of new components.
Performance. The Meta-Environment is not primarily intended for large scale industrial or commercial use. However, the performance should be such that experiments with real source code of industrial size (millions of lines of code) are possible.
Testability. It should be possible to test individual components separately.
We pay less attention to other important quality attributes such as availability and security, but we expect that they will become more important as the system evolves.
Figure Mapping between elements and views keeps track of which elements are used in which views.
|Errors & warnings||x||x|
|Graphical user interface (GUI)||x||x||x|
|Parse table generator||x||x||x|
|The Meta- Environment||x||x||x||x||x||x|
Figure End-user view of The Meta-Environment shows how simple The Meta-Environment is for an end-user. Via a graphical user interface, the user can edit ASF+SDF modules that describe the syntax and semantics of some user-defined language, let's say Java. The ASF+SDF modules will define the grammar of the Java language (this is actually already provided in the ASF+SDF library) and some functions on Java programs such as extracting facts or describing transformations on the Java code. The user can also edit an input term, that is a text that conforms to the definitions in the ASF+SDF specification (say, a Java program with a transformation function applied to it). Via the user-interface the user can now activate the application of the equations in the ASF+SDF specification to the input. The answer is an output term that will consist -- in this example -- of a transformed Java program. Of course, errors may occur and they are shown in the GUI. As an aid for power-users some additional information like compiled ASF+SDF specifications and parse tables can be exported.
The user has the following options for variability:
The filesystem directory from which The Meta-Environment is started (using the command asfsdf-meta) determines the working directory to be used during the session with the system.
The file configuration file
(in the current working directory) can be used to extend the
behaviour of the sytem. This is further explained in Extension
Points of The Meta-Environment. (LINK)
The GUI provides some options to change the overall graphical appearance (skinning) and the way in which graphs are displayed.
In the end-user view, we try to make life as easy as possible for the user and try to automate whatever we can. For instance, when a user reduces the input term to the output term, he needs not be aware of the fact that first a parse table has to be generated and that the input term is then parsed. The fact that after editing an ASF+SDF module, it may be necessary to regenerate the parse table is also implicit. The end-user is shielded from all internal representations.
The Meta-Environment mostly satisfies the design goal of usability. Some known usability weaknesses are:
The error messages generated by the parser are poor.
There is no support for finding errors in syntax definitions.
Certain features found in other IDEs are still missing (e.g., autocompletion, context-dependent help menus, and others).
The user documentation is still not yet complete and up-to-date, but we are working on this ;-)
This view does not provide any details on the actual operation of the user-interface. Some details are helpful:
The user-interface is a sovereign application, i.e., it can occupy the full window of the desktop.
The user-interface provides a graphical view on the import relations between ASF+SDF modules and the parse trees of selected modules or terms.
The user-interface provides a debugging view on the execution of ASF+SDF specifications.
This view only shows one input term and corresponding output term. In reality, several input terms may be edited and be reduced to the corresponding output term.
The decomposition of the system in parts is shown in figure The decomposition view. It emphasizes the parts that are needed to transform an input term to an output term. Not shown are the interaction with the user-interface, the precise processing of the ASF+SDF modules, and the generation of compiled specifications and parse tables.
Identical to the context diagram of the end-user view.
In addition to the ASF+SDF modules that determine the characteristics of the languages that are being defined, the user can override the default pretty printing rules for each language by giving extra ASF+SDF modules containing specialized pretty printing rules.
This system decomposition is rather traditional. It distinguishes parser, semantic operations and prettyprinter. For reasons of uniformity, pretty printing rules are also specified in ASF+SDF.
The following assumptions have been made:
Full parse trees (e.g., parse trees containing all structural and textual information, including layout and comments, of the original source text) are the best intermediate representation for programs. This is motivated by the need in software renovation projects to perform source code transformations that preserve as much of the original source text as possible.
All extra information that has to be attached to parse trees can be expressed as annotations to that parse tree. This regards source code coordinates, information about constructors, and the like.
Various simplifications have been applied in this view:
The graphical user-interface is not shown.
We abstract from the internal processing of ASF+SDF specifications.
The various internal formats that play a role are not shown.
The compilation of specifications is not shown.
The generation of parse trees (for external) use is not shown.
Figure The coordination view shows the interaction between the components of The Meta-Environment via the ToolBus middleware layer. This view can be separated in three parts:
Kernel: components providing basic functionality, like user-interface and editing.
SDF: all components related to SDF and syntax analysis.
ASF: all components related to ASF and term rewriting.
The connections between the components are established by the Tscript that is executing in the ToolBus. Note that multiple instances may exist of some components. For instance, a separate instance of the text editor exists for editing different terms or modules.
ASF checker, ASF compiler, ASF operations, ASF interpreter, ASF+SDF checker, Default prettyprint rules, Errors & warnings,GUI, Input term, Module manager, Output term, Parser, Parse table, Parse table generator, Prettyprinter, SDF checker, SDF operations, Structure editor, Text editor, ToolBus.
This architecture was designed with variability in mind. By replacing the Tscript in the ToolBus, largely different applications can be built.
The rationale for this architecture is to:
Decouple the components as much as possible.
Enable the composition of components written in different languages.
Enable the execution of components in a distributed fashion on more than one computer.
Enable variability by allowing the composition of existing and new components in new ways.
It is fair to say that the above design goals have been achieved. Components become less aware of their context and are more easily composable. Unavoidably, the Tscript has to describe all these compositions and becomes the central information hub. In large applications like The Meta-Environment these Tscripts become quite large and complex, but they are fortunately the only location where this composition information resides.
The assumptions in this view are as follows:
Components can be written in different programming languages.
The information exchange between components and ToolBus is by way of ATerms.
For all relevant languages adapters exist to connect programs written in those languages to the ToolBus.
It is possible to execute components on different, connected, physical computers. In some situations firewall settings may prevent this.
As the coordination view explained, the main coordination architecture of The Meta-Environment is provided by the ToolBus. The ToolBus runs a ToolBus script containing many processes that coordinate the features of The Meta-Environment. This Detailed Coordination View highlights these processes. These are the major coordination processes:
Editor management - the process of editing files, which includes triggering several tools that provide feedback to the user.
Module management - the process of managing modules, builds and dependencies, which includes triggering tools that provide feedback to the user.
Configuration management - the processes of managing the configuration of The Meta-Environment in terms of the visible menu options and other keyboard and mouse user interactions.
Transaction management - a generic mechanism to define critical sections that lock the use of certain available data and tools to prevent racing conditions
Note that the main coordination processes of The Meta-Environment are designed to be language independent. They reside in the meta package which is a part of the kernel layer. Other packages that depend on the meta package, such as sdf-meta (SDF layer) and asfsdf-meta (ASF+SDF layer), bind configuration parameters and specialize the generic functionality to obtain programming language specific IDE's. The way that the language specific layers extend the kernel layer is governed by this architecture description.
The common scheme of all ToolBus script processes is that there are 'listener', 'util', 'action' and 'other' processes. They are grouped into files with a straightforward file naming scheme.
Idefs are ToolBus processes that manage the connectivity between a specific tool with the ToolBus. Each idef describes exactly when and how other processes may interact with its tool, and when and how the tool interacts with the ToolBus. The abbreviation "idef" stands for interface definition.
Listener are processes that catch notes that are send by mostly the idef of the module manager and sometimes by utility processes. The listeners are a facade only, they forward to util processes. Listener files begin with a language name or a feature name and end with "-listeners.tb"
Actions are processes that are triggered directly by a user interaction, which forward to util processes. Files begin with a language name or a feature name and end with "-actions.tb"
Utils are the implementation processes, they do the actual work. Files begin with a language name or a feature name and end with "-utils.tb"
Note that communication can be done in ToolBus scripts in three ways: sending messages, sending notes and calling processes. In general the rules communication of The Meta-Environment management processes are:
tool processes - which are single processes each associated with a single tool, also called 'idefs' - send and receive synchronous messages, except for manager tools which can also broadcast asynchronous notes.
notes are received by listener processes, which call util processes
utilitity processes call eachother (synchronously), and send messages to tool processes, where it is tried to wrap single messages to tools in simple process definitions
The following picture provides an overview of the kinds of processes and their interaction with eachother and tools. The module manager is just a tool, but it is very central to the scheduling of all work in the Meta-Environment, so it has been singled out. See also the following section on module management.
The goal of the editor manager is to manage the complexity of appearing and dissappearing editors. An editor in the Meta-Environment is the pair of a visual text editor, implemented in Java, and a structural tree 'editor' implemented in C. Editors are data-sources for several (interactive) usage scenarios in the Meta-Environment. Because they appear and dissappear when the user opens or closes files, this is where the first racing conditions start to occur in The Meta-Environment. The editor-manager pre-dates the transaction management system and implements it's own kind of transactions to lock resources that are associated with editors.
There are four ToolBus tools involved:
editor-manager - a C tool that maintains a registry of editing processes. Each process is identified with a unique id. The creation and garbage collection of editors is managed here. Also this tool provides the binding between a text-editor instance and a tree that is stored in the structure-editor.
structure-editor - a C tool that stores parse trees with editor id's and supports a number of interesting queries on these trees. The tool actually is not an editor in the sense that it never changes a tree
addPosInfo - the editing infra-structure highly depends on the fact that all parse trees are passed through this tool before anything else happens
editor-plugin - a Java program that uses the Swing editor kit to provide visual editing of source files
Most of the coordination work is implemented in ToolBus scripts:
editing.tb - provides generic and reusable processes
<lang>-editor-utils - binds the parameters of editing.tb processes and makes them language specific. This is where it is decided what happens when the user clicks the mouse or activates a pop-up menu.
There is always at least one ToolBus process per file edited, which stays alive until the last transaction as declared using the editor-manager has ended. This means that although an editor is not visible on the screen anymore, the data stored with it in Java and in the structure-editor is available until the transaction ends.
The goal of module management is to trigger functionality (utils) on source files, such as .sdf or .asf files, at the right time and to provide the user with a consistent view of the state of the source files that are edited by the user in the environment.
The module-manager tool connects to the Bus and maintains an administration of which modules are currently loaded and what their dependencies are
The module-manager maintains a list of key-value attributes (ATerms) for each module. Each key has a namespace and a name. Any T-script can set or change the attributes of a module.
The module-manager broadcasts notifications on the ToolBus whenever a module is added or removed, a dependency is added or removed, or an attribute is changed.
Language specific scripts in the SDF or ASF+SDF layer register 'event rules' with the module-manager. The event rules define how some attributes on modules are set automatically by propagating information across the dependency graph between modules. This event rule mechanism can deal with cyclic dependencies. A single change of an attribute on a module, can thus spawn a number of changes on other modules, all changes are broadcasted as independently on the ToolBus.
Event listeners (note receivers in T-script) listen to the changes in the module manager and trigger appropriate utils. Some utils change the attributes of modules, triggering other utils via other listeners.
There is one central attribute that is used a lot by The Meta-Environment called "status". A particular view on The Meta-Environment module management process is that it is a collection of state machines, one per each module:
The status attribute of a module can have a fixed set of values, called the "states" of the module.
Specific processes are tied to states by the event listeners. When the status attribute changes to a particular state, the processes that belong to that state are executed using listeners
Processes that are finished set the next state of modules.
Like any other attribute, the state attribute can also be updated by event rules, such that modules change to a different state without ToolBus script intervention. Examples of common state values are
unknown - for modules that somebody depends on, but nothing is known about yet
identified - for modules that have been located on disk and have been given a name and a path attribute
error - for modules that contain errors that need to be solved by the user
child-error - for modules that depend on modules that have errors that need to be solved by the user
complete - for modules that are in an executable state
See fig. the detailed coordination view on the role of the module manager in its environment. As an example, consider the scenario in which a user saves an SDF file in The Meta-Environment:
The editor is a GUI tool, which will send an event to the ToolBus, containing for which language (SDF), and which file the event is meant for
The idef of the editor will receive the event and call the action process that deals with SDF file saving
The action process will first find out to which module the file belongs to by querying the 'path' attribute in the module manager. Then it will call the appropriate utility process to do the work.
The utility process coordinates everything that needs to be done for the current module. It works by activating tools such as the in-output tool which can save files to disk. When the work for file saving is done, the utility process will also update the status attribute in the module manager by sending a message to the idef of the module manager
The idef of the module manager relays the updated status attribute value to the module manager
The module manager receives the updated status attribute value and - starting from the initial change and using the dependencies between modules - it will change the status attributes of other modules. The changing is governed by several attribute update rules.
For every changed status attribute for a specific module, the module manager sends out an event
The event is caught by the idef of the module manager, which broadcasts the updated status attribute values information using a ToolBus note
Several listener processes may be subscribed to such status notes. One example is the SDF module parser listener. This listener will trigger the SDF parsing utility process for a specific module.
The SDF parsing utility process, and many other processes triggered by other listeners are now working in parallel as in Step 4. Some utility processes may send messages to GUI tools, for example the new tokens to highlight are send to the editor tool from Step 1.
Steps 4 to 10 will repeat until a fixed point is reached and no more attributes in the module manager change, or no more listeners react to the changes in attributes.
Utility processes should generally deal with only one module at a time. The distribution of work for several modules should be scheduled by attribute event rules and the notes that are send out by the module manager because of these rules. If a utility process asks for the dependencies of a module, then this is a sign of code that goes agains this architecture.
After the initial attribute change, the listeners will trigger many instances of many processes which will run mostly in parallel and asynchronously. This is when racing conditions can occur and transaction management starts playing its role.
Although preferences and other kinds of start-up and run-time
configuration parameters have been factored out in the config-support
and configuration-manager packages, they have not been generalized or
parameterized. These two packages support a fixed set of parameters
that is specific for the ASF+SDF Meta-Environment. Which configuration
options are supported is define in
this functionality resides in the core of the Meta-Environment, it is
explicitly linked with features in ASF+SDF. This needs to be fixed in
the future, because it is hard to know where and how to extend this
implementation to add new configuration parameters.
Configuration management works as follows:
At start-up time, the Meta-Environment ToolBus script read
.actions files from disk. One contains
the default bindings of the parameters of all tools in The
the other is a
.actions files in the users
workspace directory. The files are registered with the
configuration-manager, which resolves import statements and
maintains a hierarchy where user preferences have precedence over
default system preferences.
Several utility processes distribute the configuration information on demand to the tools that need it. These processes first query the configuration manager for information (using cm-... messages), and then forward this information to the appropriate tools. For example, when an editor is started, the editing scripts ask the config-manager what kind of menu's need to be shown. This information is forwarded to the editor. The editor tool will receive an ATerm encoding of the menu structure, which is interpreted to produce the right menu's.
Some configuration parameters define what needs to be done and when. Since this is a responsibility of the ToolBus scripts there is a bridge. Such configuration parameters carry the names of processes to run. The idef of a tool that interprets such a configuration parameter will call the appropriate process using a dynamic process call. For example, the MetaStudio knows which menus to show. Each menu option will have an associated Process name. When the menu option is selected by the user the MetaStudio sends an event to its idef, which calls the process using a dynamic call.
Due to the design of the module manager and the scheduling of
processes via listeners, many things in the Meta-Environment happen
asynchronously. Still the processes share common data sources which
they read and write to. The transaction management system is based on
simple locking. For each shared resource the
transactions.tb script allows to start a
TransactionManager process. A lock is obtained by sending a message to
this unique process, which only allows the next process to obtain a
lock until the previous has released it.
This system is completely incremental and demand driven. Processes are not obliged to obtain a lock to obtain access to shared resources. Instead, when the programmer knows that his process could lead to racing conditions, he decided to first obtain a lock to the resource. Processes that are robust against changing values, such as progress views, do not lock the resources they read from, which improves efficiency by allowing parallelism.
For each of the main coordination processes there are different variability stories
Each editor has a type name to identify the language it is for.
It therefore can behave differently by getting different configuration
parameters, and show different menus and do different things for mouse
actions. Also, each type of language has a different editing process,
which extends the functionality of
with listeners that act differently for every language.
The generic nature of the module manager makes it easy to adapt and extend it to new programming languages.
Add a new 'name space' to add a new language
Add a new 'event rule' to propagate new information acros the dependency graph
Add a new note listener to add new automatic features to the environment.
The configuration manager has all parameters more or less hard
wired. They are generated from the
basic/Config.sdf file. The manager also knows
about specific parameters, taking care for example that menu options
are treated in order.
The transaction mechanism is totally orthogonal and can be extended with new kinds of locks using a ToolBus script one-liner. Note however that adding new requests for existing locks always has the possibility of introducing deadlock. This occurs naturally when requests for locks of the same resource are nested in a single scenario. It can also happen if ToolBus scripts accidentally become dependent on eachother in a circular fashion. Care must be taken to fully understand which resources are needed and when.
In other words, adding locks to the ToolBus scripts of The Meta-Environment is not a modular activity. One must fully understand all dependencies between processes and shared resources to make sure this is done correctly.
In general, the design rationale behind the processes in The Meta-Environment is reusability. The main processes are to be designed in such a way that they are programming language independent. The goal is to be able to instantiate an IDE by providing the right parameters to the generic ToolBus scripts to obtain language specific IDE behavior.
The other main rationale behind these designs is extensibility. For example, the asynchronous, parallel and independently addable event listeners make sure that the system can be layered, and also extended without changing the existing functionality. However, this does introduce the need for transaction management.
For traceability of ToolBus scripts we try to use ToolBus process calls over snd-messages whenever possible, except to communicate with tools. This allows the ToolBus type checker to find common mistakes early in the process. Also the trace of process calls is more easy to get an overview of than the trace of messages and notes, since every process call has a single caller and a single callee. Messages on the other hand are send by one message but could be received by anybody.
The configuration management process of The Meta-Environment is not general enough to be easily reusable or extensible. There is still a lot of editing in existing source code involved when reusing or extending it.
The focus on reusability and extensibility has introduced strains on the ToolBus in terms of efficiency. Specific optimizations that help managing large volumes of notes, messages and process calls have helped a lot in making the systems run more smoothly:
Statically computing and dynamically maintaining a map that finds possible communication partners for both messages and notes, as opposed to looping over all available processes.
Changing the use of highly generic snd-msg's for binding configuration parameters to the use of dynamic process calls. The benefit is that there is no need to find a communication partner since the process is already named.
The Meta-Environment uses ATerms as pervasive data format to store and exchange data. On top of the ATerms, various specialized data formats have been defined as shown in figure The data layer view.
New data formats can be added by writing new data definitions (using ADT) and generating the interfaces for these data using APIGEN.
The fundamental assumption is that tools exchange data in the form of ATerms. Since specialized services require specialized data formats, it makes sense to build them on top of ATerms.
The above data formats have been added out of necessity. It is likely that more data format will be added for:
Rstores: (name, relation) pairs as used by Rscript.
An issue that requires attention is whether the ToolBus should be aware of these datatypes. Currently, it just pumps ATerms around, but more specialized typing of the tool interfaces would help implementors with early error detection.
For reasons of modularity, reuse and maintenance, the implementation of The Meta-Environment is subdivided in packages. A package is the smallest unit of software that can be distributed or re-used. Currently, The Meta-Environment provides over 60 packages that provide a wide range of functionality. The table Mapping between elements and packages shows how the architecture elements in the various views are implemented by one or more packages. Note that some packages (e.g., aterm, aterm-java, shared-objects, JJTraveler) are used by most package; they are not listed in the table.
|ASF compiler||asf, asc-support|
|ASF+SDF library||asf-library, sdf-library|
|Configuration manager||config-manager, config-support|
|Errors & warnings||error-gui, error-support|
|Graphical user interface (GUI)||dialog-gui, graph-gui, graph-support, meta-studio, module-details-gui, navigator-gui, progress-gui|
|Module manager||module-manager, module-support|
|Output term||pt-support, pandora|
|Parse table generator||pgen|
|Term store||term-store, io-support|
|The Meta-Environment||asfsdf-meta, sdf-meta, meta|
|ToolBus||toolbus, toolbuslib, toolbus-java-adapter|
The following variability issues have been identified.
All packages have been written with portability in mind.
We use configuration tools (autoconf) to adjust packages to local installation requirements.
All software development has been done on Linux. However, the majority (but certainly not all) of the packages have been compiled successfully on Windows as well.
The reasons for splitting up the system in so many packages is the following:
Each package represents a clearly defined functionality. This localizes the impact of future modifications.
Each package can be reused independently.
The package structure makes explicit dependencies between packages mandatory. This makes it easier to study the impact of more global changes.
On the positive side, we do have achieved an incredibly modular and flexible architecture. On the negative side, we do have to admit the following:
Although "all packages are considered equal" in our approach, in reality this is not the true. A handful of primary packages (asfsdf-meta, aterm, pandora, pgen, sglr, shared-objects, toolbus) are frequently downloaded and reused in a variety of applications, the others are just implementation details for those primary packages.
Initially, we exposed the flat list of all our packages to the outside world. This turned out the be very confusing.
In some cases, a more monolithic structure for subcomponents would make the life of implementors somewhat easier.
There are some costs involved in creating and maintaining a new package. We are therefore continuously simplifying and optimizing our build environment to reduce these costs.
Despite these critical notes, we do think that the overall package approach is highly effective.
We have made the following assumptions that are mostly hardwired in this architecture:
Most packages can read/write data in the form of ATerms or any of the dataformats built on top of ATerms.
Most packages provide their functionality in three forms;
As command line tools.
As a library that can be called from other programs.
As a ToolBus tool that can be connected to a ToolBus-based system.
The dependencies between the packages of the Meta-Environment are shown in figure The package dependency view. Clearly, four separate layers can be distinguished (from bottom to top):
The infrastructure layer that provides common facilities.
The kernel layer that provides a kernel meta-environment.
The SDF layer that provides a meta-environment for editing SDF specifications.
The ASF layer that provides the full ASF+SDF Meta-Environment.
In order to reduce the visual clutter in this figure, various simplifications have been applied:
All uses by elements in the top layers of facilities in the bottom layer are summarized by one arrow: "may use any of".
An element uses all other elements to which it is directly or indirectly connected.
This package structure has been designed to accommodate variability; it enables arbitrary combinations of packages. This variability can be achieved by building new packages that depend on combinations of existing packages.
The design rationale for this package structure is the following:
We come from a situation in which many packages had dependencies on ASF+SDF. We considered this undesirable, from the perspective of modifiability and extensibility.
By organizing the package dependencies in layers, we achieve an architecture that can be easily extended. Starting with a widely used infrastructure layer, we first provide a kernel meta-environment that has a user-interface and basic editing facilities. Next, we add an SDF layer that provides syntax definitions via SDF, parser generation and parsing. Finally, we add ASF to provide semantic definitions and term rewriting.
Each of these layers can form the starting point for the implementation of systems with widely varying functionality.
The fact that various "clones" of the Meta-Environment exist proves the success of this approach.
The primary program representation is the parse tree (using AsFix). This assumptions is becoming more and more true in the context of software analysis, program transformation and software renovation. However, in the context of ordinary compilation this maybe overkill and lead to inefficiencies.
All packages need to use the same build interface in order to cooperate in this package structure.
For the record, there are some issues that should be resolved in this view:
Only the dependency on graphviz is made explicit in the package definitions.
The mutual dependency between tide and meta-studio is curious and should be investigated.
It is a pity that there are still dependencies between the "meta" layer and asc-support. This means that not yet all ASF dependencies have been cut out of the kernel meta-environment.
Running The Meta-Environment is currently dependent on the following external packages:
Power-users can use the system in four ways:
By using the GUI.
By using some of the commands.
By writing programs and using the provided libraries.
By modifying or extending the standard ToolBus scripts that are provided.
In advanced applications all three ways will play a role. This can be used to implement completely tailored Meta-Environments.
The power-user is also confronted with the various data formats that are used: ATerms, AsFix, Parse tables
Table 1.4. Main commands
|pt-dump||Module name (String)||Parse table (ATerm)|
|sdf2table||SDF definition||Parse table (ATerm)|
|eqs-dump||Module name (String)||Equations (AsFix)|
|asfe||Equations (AsFix)||Parse tree (AsFix)|
|sglr||Parse table (ATerm), String||Parse tree (AsFix)|
|apply-function||Function name (String), Sort name (String), Module name (String), Term (AsFix)||Term (AsFix)|
Table 1.5. All commands and libraries per package
|Package||Summary||Commands||Libraries||C includes (C), Tscript (T), ASF+SDF (ASF), Abstract Data Type (ADT)|
|apigen||API generator for C and Java|
|asc-support||Run-time library used by compiled ASF+SDF specifications.||libasc-support-me.a||C|
|asf||All tools for implementing ASF||ASFSDFApigen addeqssyntax asfc asfchecker asfe asfoperations asfsdfchecker concat-asf lift-asf||T|
|asf-library||Source library of ASF+SDF definitions||ASF|
|asf-support||Support libraries for ASF tools||libASFME.a||C|
|asfsdf-meta||The ASF+SDF Meta-Environment||asfsdf-meta eqs-dump||T|
|aterm||ATerm library and supporting tools||atdiff atrmannos atsum baf2taf baf2trm baflle dicttoc taf2baf taf2trm termsize trm2baf trm2taf trmcat||libATerm||C|
|aterm-java||Java version of ATerm library||aterm-1.6.jar|
|config-support||Support library for configuration manager||libConfigAPI.a, configapi.jar||C|
|dialog-gui||GUI component for dialogs||dialog.jar||T|
|editor-manager||Editor manager supervises concurrent editing sesssions||editor-manager||libEditorManager.a||T|
|editor-plugin||Java embeddable editor component||editor-plugin-1.0.jar||T|
|error-gui||GUI components for error display||error-viewer-1.0.jar||T|
|error-support||Tools for handling errors (Summaries)||error-diff error-support lift-error lower-error||libErrorAPI.a libLocationAPI.a errorapi.jar locationapi.jar||C,T|
|graph-gui||GUI component for graph display||graph-painter.jar prefuse.jar||T|
|graph-support||Tools for handling graphs||graph2dot layoutgraph tree2graph||libGraph.a graph-0.2.jar||C, T, ADT|
|io-support||ToolBus tool for file i/o.||in-output||libIOAPI.a ioapi.jar||C, T, ADT|
|meta-studio||Main GUI component||gui||gui-util.jar gui.jar idw-gpl-1.4.0.jar||T|
|module-details-gui||GUI component for display detailed info about modules||moduledetails.jar||T|
|module-manager||Manages a collection of modules||module-manager||module-manager-1.0.jar||T|
|module-support||Tools for handling modules||moduleapi.jar|
|navigator-gui||GUI components for navigating through module hierarchy||navigator.jar||T|
|pgen||Parse table generator||parse-sdf2 parsetablegen removevarsyntax sdf2table sdfchecker||T|
|progress-gui||GUI component that display progress of an operation||progress.jar||T|
|pt-support||Support libraries for parse trees||addPosInfo ambtracker apply-function comparePT flattenPT implodePT liftPT unparsePT unparseProd||libPTMEPT.a libmept.a||C, T|
|sdf-library||Source library of SDF definitions||ASF|
|sdf-meta||An SDF-only Meta-Environment||def-dump pt-dump sdf-meta||T|
|sdf-support||Support libraries for handling SDF definitions||sdf-modules sdf-renaming||libPT2SDF.a libSDF2PT.a libSDFME.a||T|
|sglr||SGLR parser||dump-actions dump-gotos dump-priorities dump-productions restorebrackets sglr||libsglr.a||C, T|
|shared-objects||Library for maxiamlly shared objects in Java||shared-objects-1.4.jar|
|structure-editor||Tools for syntax-directed editing||structure-editor||libStructureEditor.a||T|
|term-store||Generic store for terms||term-store||T|
|tide||Generic debug framework||tide tide-gdb||tide.jar||T|
|tide-support||Tools for connecting C-based tools to tide||libtide-adapter.a||C, T|
|toolbus||ToolBus coordination architecture||bc-adapter emacs-adapter idef2tif merge-tifs start-emacs tblog toolbus uri-encode ctif gen-adapter informer perl-adapter tbgraph tifstoc toolbus-adapter wish-adapter||C|
|toolbus-java-adapter||Adapter for coupling of Java to ToolBus||java-adapter tifstojava||toolbus-java-adapter-1.0.jar|
|toolbuslib||Library for C-based ToolBus tools||tbunpack||libATB.a||C|
|JJTraveler||Traversal pattern in Java.||jjtraveler-0.5.jar|
There are two versions of The Meta-Environment. One has a home grown GUI and GUI extension framework (MetaStudio), the other is based on and fully integrated with Eclipse using IMP (http://www.eclipse.org/imp). This view describes the differences between the two instances. The goal of this view is to help the designers and implementers of the Eclipse based Meta-Environment make design decisions.
The main thing to notice is that both environments have the same high-level architecture: the ToolBus is the central mechanism for coordination. However, as Eclipse already integrates and composes several tools and does not use the ToolBus there may be overlapping functionality, or functionality from Meta-Environment needs to be replaced by or ported to Eclipse.
The ToolBus design allows us to maintain a heterogeneous implementation of The Meta-Environment. This allows phased evolution and separation of concerns, at the price of integration which is mitigated by the ToolBus.
IMP builds on Eclipse, and in particular on Eclipse's plugin mechanism that uses the concept of extension points. Read about extension points at eclipse.org. It is important to understand OSGI and Eclipse's plugin mechanism in some detail.
IMP revolves around the concept of a language descriptor, which identifies the name of a language using an identifier and the associated file extensions. A language descriptor is declared using an extension point.
IMP services are predefined interfaces that can be implemented for every language. The binding between an implementation of such an interface and a language is done using an extension point.
IMP uses the implementation of services to orchestrate IDE features of Eclipse. The central orchestration is done in the "UniversalEditor" class. Other orchestration is done via the Eclipse Nature and Builder facilities.
In our case, the languages we are talking about are SDF and ASF+SDF. So, we could instantiate IMP by implementing the IMP services specifically for these languages. This is not the way Meta-Environment is designed. As said in other views, there is a kernel layer which contains both language independent and language parametric functionality. The goal of a PIMP'ed Meta-Environment is to bridge Meta-Environment to IMP at this kernel layer in such a way that we can easily use Meta-Environment to get other IMP-based IDE's. We expect for example to produce IDE's for RScript and newer version of ASF+SDF. Therefore, we need language parametric implementations of all IMP services.
A language that builds on the Meta-Environment kernel, such as SDF, will declare an extension point for each service that points to the language parametric service implementation.
This service implementation will (in 99% of the cases) be a ToolBus tool (a facade basically), that forwards the work to a ToolBus script.
The ToolBus script implements the features that the service needs and communicates information back to the language parametric service implementation class, and/or activates other tools.
Note that for each IMP service there will be
a Java class that both is a ToolBus tool and implements the IMP service interface
a ToolBus idef process that communicates with this tool integrates with the rest of the Meta-Environment coordination scripts
Examples of IMP services that are ToolBus tools are:
IParseController - is called when a file needs to be parsed according to the UniversalEditor, the IParseController implementation relays this message to the ToolBus, which calls sglr, etc. etc.
ITokenColorer - is called when the UniversalEditor decides it is time to add highlighting to the editor. The ITokenColorer implementation receives a handle to the UPTR parse tree (an ATerm), sends this to the ToolBus to be processed by the structure editor and receives a list of source locations tupled with font names back. Then it returns the TokenIterator which is requires by the UniversalEditor which iterates over the source locations.
IMP does not hide all of Eclipse. Especially concepts such as projects, resources, and error markers & annotations originate from Eclipse.
To communicate with these Eclipse features, small tools that bridge to the ToolBus should be created, and data that are interfaces and classes in Eclipse should be reifed as ATerms. This is completely parallel to the implementation of IMP services. The difference is that the number of IMP services is a closed and small set, while Eclipse features are open-ended, especially considering the pletora of Eclipse plugins.
In many cases, the Meta-Environment has similar concepts that only need to be adapted to fit with Eclipse IMP. One example is errors. The error ATerms can be registered as Eclipse error annotations by a simple adapter written in Java that is connected to the ToolBus.
Some concepts have "impedence mismatch". The concept of an Eclipse resource closely resembles the Meta-Environment concept of a module. However, the two are actually rather different. The module concept of the Meta-Environment needs to be adapted to fit the resource concept of Eclipse. The end result of the PIMP'ed Meta-Environment should be something that satisfies the expectations of Eclipse users.
Note that each Eclipse feature that is reifed as a ToolBus tool will have:
A Java class that is a ToolBus tool which takes ATerms from the ToolBus an gives them a meaning by calling Eclipse functionality.
A ToolBus idef process that communicates with this tool and integrates with the rest of the Meta-Environment coordination scripts.
Examples of tools that reify Eclipse features are:
org.meta_environment.eclipse.errors.ErrorViewer: it implements the same idef as the Meta-Environment error viewer, receiving lists of error messages. The implementation of the tool simply registers the errors with the Eclipse annotation manager
org.meta_environment.eclipse.files: communicates all identified resources and registers them with the module manager.