:sectnums:
== Goal

A mechanism is needed by which programs executed under the Java^TM^ virtual
machine but written in languages other than the Java programming language,
can be debugged with references to the original source (for example, source
file and line number references).

Constraints:

* No change to the Java programming language.
* Optional change to Java programming language compiler.
* No change to JPDA clients (debuggers, ...) for basic functionality. With the
  exception being, tools that arbitrarily prohibit non-Java programming language
  source.
* Minimal change to Java virtual machine.
* No change to the Java platform class libraries.

== Terminology

[width="100%",cols=".^30%s,70%",options="header"]
|===
|Term |Definition
|[#TermFinalSource]#final-source#
|The final form of source. This source will be <<TermCompiler,compiled>> into a
<<TermClassFile,class file>>. Typically, final-source is Java programming
language source.

|[#TermTranslatedSource]#translated-source#
|Source that will be translated by a
<<TermLanguageProcessor,language-processor>> into another language or form.

|[#TermLanguageProcessor]#language-processor#
|Converts <<TermTranslatedSource,translated-source>> to
<<TermFinalSource,final-source>> or to different
<<TermTranslatedSource,translated-source>>.

|[#TermClassFile]#class file#
|A collection of bytes in Java virtual machine class file format.

|[#TermCompiler]#compiler#
|Compiler which converts <<TermFinalSource,final-source>> to class files. For
example, `javac` is a compiler.

|[#TermPostProcessor]#post-processor#
|Takes a <<TermClassFile,class file>> and a
<<TermSourceMapFile,Source Map File>> as input; generates a class file as
output. Inserts a <<TermSourceDebugExtension,SourceDebugExtension attribute>>.

|[#TermSourceMap]#Source Map# +
SMAP
|Specifies the mapping of source between one or more sets of input source and
the output source. See: <<Source Map Format>>.

|[#TermSourceMapFile]#Source Map File# +
SMAP-file
|Specifies the mapping created by a
<<TermLanguageProcessor,language-processor>> between
<<TermTranslatedSource,translated-source>> input and the resultant output
source. It is a <<TermSourceMap,Source Map>> stored in a file in
<<Source Map Format>>.

|[#TermStratum]#stratum#
|A view of a class from a particular, named, programming language level.

|[#TermJPDA]#JPDA#
|The https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/index.html[Java
Platform Debugger Architecture].

|[#TermJDI]#JDI#
|The Java Debug Interface, which is the high-level Java programming language
interface of the
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/[Java Platform Debugger
Architecture]. See:
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/[JDI Specification].

|[#TermSourceDebugExtension]#SourceDebugExtension +
attribute#
|A Java virtual machine class file attribute that holds information about the
source. In this case, the information is a
<<TermSourceMap,Source Map>> in <<Source Map Format>>. The source
is mapped to <<TermFinalSource,final-source>> (output source). The Source Map
has potentially multiple sets of input source (<<TermStratum,strata>>).
See: <<attribute,SourceDebugExtension Class File Attribute>>
|===


== Approach

=== Single Translation

A <<TermLanguageProcessor,language-processor>> translates
<<TermTranslatedSource,translated-source>> to <<TermFinalSource,final-source>>
(the case of translation from one <<TermTranslatedSource,translated-source>> to
another translated-source, is addressed <<Multiple Translations,below>>). It
also creates a second output, an <<TermSourceMapFile,SMAP-file>>, whose format
is described in <<Source Map Format>>. This file describes the mapping between
input source and output source (e.g. line number and source file).

The <<TermFinalSource,final-source>> generated by the
<<TermLanguageProcessor,language-processor>> is compiled by the
<<TermCompiler,compiler>>.

The <<TermPostProcessor,post-processor>> takes the class file generated by the
<<TermCompiler,compiler>> and the <<TermSourceMapFile,SMAP-file>> as input. A
<<TermSourceDebugExtension, SourceDebugExtension>> attribute containing the SMAP
in the SMAP-file is added to the class file and the new class file is written.

Optionally, the <<TermCompiler,compiler>> may take both
<<TermFinalSource,final-source>> and the SMAP-file as input, and perform both
compilation and installation of the SourceDebugExtension.

When the resultant program is debugged using a debugging tool based on the Java
Debug Interface +
(<<TermJDI,JDI>>) of <<TermJPDA,JPDA>>, the final-source line
number information is converted to the specified language view
(<<TermStratum,strata>>).

=== Multiple Translations

A language-processor might translate source into source which will become input
to another language-processor, and so on. Eventually, after possibly many
translated-source forms, final-source is produced. Each translation produces
SMAP information. This information must be preserved and placed in context, so
that each <<TermStratum,stratum>> can be mapped to the final-source.

A language-processor checks for an <<TermSourceMapFile,SMAP-file>> in a location
parallel to that of the input source. For example, if the source repository is a
file system and the input source is located at `path name.extension` then
`path name.extension.smap` will be checked for an SMAP. The input SMAPs will be
copied into the generated SMAP. See <<Embedded Source Maps>> for specifics on
the embedding of input SMAPs.

In the case of multiple translations, the <<TermPostProcessor,post-processor>>
must resolve the embedded SMAPs. See <<SMAP Resolution>>.

Note that final-source need not be Java programming language source, as
compilers for other languages may directly generate class files, including the
`SourceFile` and `LineNumberTable` class file attributes. SMAPs and the
mechanism presented here are still useful for handling multiple translations.

A programming language implementor, directly generating class files might also
choose to generate SMAPs (thus functioning as both language-processor and
compiler) since the SMAP is useful for describing source configurations (such as
multiple source files per class file) which cannot be represented with the
`SourceFile` and `LineNumberTable` attributes. In this case, the input is
translated-source and the final-source is represented in the attributes but is
never generated.

=== Diagram

This diagram demonstrates data flow. The particular case shown has two levels of
translation, with file inclusion on the second level (as is the case in the
example in <<SMAP Resolution>>).

image:data-flow.png[data flow diagram,pdfwidth=75%]

Where `TS` is <<TermTranslatedSource,translated-source>> and `FS` is
<<TermFinalSource,final-source>>.

== Scope

=== Variables

The complexity of mapping semantics (like variable and data views) across
languages is such that this feature has not been included in this version of the
specification and will be considered for a future version.

=== Multi-Level Source View

The ability to choose the source level to view is addressed in this
specification. These are referred to as <<TermStratum,strata>>.

=== Finding Source Files

Currently, <<TermFinalSource,final-source>> is found by combining the follow
elements:

* A source path
* The package name converted to a directory path
* The source file name from a <<TermJDI,JDI>> call (derived from the `SourceFile`
  class file attribute)

Since existing debuggers use this mechanism (the only way for an existing
debugger to find <<TermTranslatedSource,translated-source>>) each aspect must be
addressed:

* The source path must be set-up to include translated-source directories
* Source must be placed in a directory corresponding to the package
* The <<TermJDI,JDI>> call must return the translated-source name.

For debuggers written against the new APIs, a new method has been added which
returns the source path - this makes the translated-source directory structure
flexible.

=== Multiple Source Files per Class File

When an inclusion mechanism is used, a class file will contain source from
multiple <<TermTranslatedSource,translated-source>> files. The SourceFile
attribute of class files only associates one source file with a class file which
is one reason the approach of simply rewriting the `SourceFile` and
`LineNumberTable` attributes had to be abandoned. The SMAP allows a virtually
unlimited number of source files per stratum.

== Source Map Format

A Source Map (SMAP) describes a mapping between source positions in an input
language (<<TermTranslatedSource,translated-source>>) and source positions in a
generated output language. A view of the source through such a mapping is called
a <<TermStratum,stratum>>. The <<TermSourceMapFile,SMAP-file>> contains an
unresolved SMAP. The <<TermSourceDebugExtension,SourceDebugExtension>> class
file attribute, when used as described in this document, contains an SMAP. The
SMAP stored in a SourceDebugExtension attribute must be
<<SMAP Resolution,resolved>>, and thus will have no
<<Embedded Source Maps,embedded SMAPs>> and will have the
<<TermFinalSource,final-source>> language as the output language.

An SMAP consists of a header and one or more sections of mapping information.

There are currently seven types of section: stratum sections, file sections,
line sections, vendor sections, end sections, and open and close embedded
sections. New section types may be added in the future - to facilitate this, any
unknown sections must be ignored without error.

The semantics of each section is discussed below. For clarity, an informal
description of the syntax of each section is included in the discussion. See the
formal <<SMAP Syntax,SMAP syntax>> for syntax questions.

=== General Format

The SMAP consists of lines of
http://www.unicode.org/unicode/standard/standard.html[Unicode] text, with a
concrete representation of http://ietf.org/rfc/rfc2279.txt[UTF-8]. Line
termination is with line-feed, carriage-return or carriage-return followed by
line-feed. Because SMAPs are included in class files, size of the SMAP was an
important constraint on the format chosen for them.

[#smap-header]
=== Header

The first line of an SMAP is the four letters `SMAP` which identifies it as an
SMAP. The next line is the name of the generated file. This name is without path
information (and thus if the generated file is final-source, the name should
match the `SourceFile` class file attribute). The last line of the header is the
default stratum for this class. The default stratum is the stratum used when a
debugger does not explicitly specify interest in another stratum. In an
unresolved SMAP the default stratum can be unspecified (blank line). In a
resolved SMAP the default stratum must be specified. A specified stratum must
either be one represented with a stratum section or `Java` which indicates the
standard final-source information should be used by default.

=== StratumSection

An SMAP may map more than one <<TermTranslatedSource,translated-source>> to the
output source (the output source is <<TermFinalSource,final-source>> if the SMAP
is in a <<TermSourceDebugExtension,SourceDebugExtension>>). A view of the source
is a stratum (whether viewed as translated-source or final-source). Each
translated-source language should have its own stratum section with a unique
stratum name. The final-source stratum (named "`Java`") is created automatically
and should not have a stratum section. The stratum section should be followed by
a file section and a line section which will be associated with that stratum.

The format of the section is simply the stratum section marker `*S` followed by
the name of the stratum. The section ends with a line termination. One
<<FileSection>> and one <<LineSection>> (in either order) must follow the
StratumSection (before the next StratumSection or the <<EndSection>>). One or
more <<VendorSection>>s may follow a StratumSection. There must be at least one
<<StratumSection>>.

=== FileSection

The file section describes the translated-source file names. Each line maps a
file ID to a source name and, optionally, to a source path. File IDs are used
only in the <<LineSection>>. The source name is the name of the
translated-source. The source path is the path to the translated-source, the "/"
symbol is translated to the local file separator. In the case where the source
repository is a file system, source name is the file name (without directory
information) and source path is a path name (often relative to one of the
compilation source paths). For example: `Bar.foo` would be a source name, and
`here/there/Bar.foo` would be a source path. The first file line denotes the
primary file.

The format of the file section is the file section marker `*F` on a line by
itself, followed by file information. File information has two forms, source
name only and source name / source path. The source name only form is one line:
the integer file ID followed by the source name. The source name / source path
form is two lines: a plus sign `+`, file ID, and source name on the first line
and the source path on the second. The file ID must be unique within the file
section. A <<FileSection>> may only occur after a <<StratumSection>>. The
`FileName` must have at least one character. The `AbsoluteFileName`, if
specified, must have at least one character.

For example:

 *F
 + 1 Foo.xyz
 here/there/Foo.xyz
 2 Incl.xyz

declares two source files. File ID #1 has source name `Foo.xyz` and source path
`here/there/Foo.xyz`. File ID #2 has source name `Incl.xyz` and a source path to
be computed by the debugger.

=== LineSection

The <<SyntaxLineSection,line section>> associates line numbers in the output source
with line numbers and source names in the input source.

The format of the line section is the line section marker `*L` on a line by
itself, followed by the lines of <<SyntaxLineInfo,LineInfo>>. Each
<<SyntaxLineInfo,LineInfo>> has the form:

[subs=normal]
----
<<SyntaxInputStartLine,InputStartLine>> # <<SyntaxLineFileID,LineFileID>> , <<SyntaxRepeatCount,RepeatCount>> : <<SyntaxOutputStartLine,OutputStartLine>> , <<SyntaxOutputLineIncrement,OutputLineIncrement>>
----

where all but

[subs=normal]
----
<<SyntaxInputStartLine,InputStartLine>> : <<SyntaxOutputStartLine,OutputStartLine>>
----

are optional.

A range of output source lines is mapped to a single input source line. Each
<<SyntaxLineInfo,LineInfo>> describes <<SyntaxRepeatCount,RepeatCount>> of these
mappings. <<SyntaxOutputLineIncrement,OutputLineIncrement>> specifies the number
of lines in the output source range; this line increment is applied to each
mapping in the <<SyntaxLineInfo,LineInfo>>. The source file containing the input
source line is specified by <<SyntaxLineFileID,LineFileID>> via the
<<FileSection>>.

More precisely, for each `n` between zero and

[subs=normal]
----
<<SyntaxRepeatCount,RepeatCount>> - 1
----

the input source line number

[subs=normal]
----
<<SyntaxInputStartLine,InputStartLine>> + n
----

maps to the output source line numbers from

[subs=normal]
----
<<SyntaxOutputStartLine,OutputStartLine>> + (n * <<SyntaxOutputLineIncrement,OutputLineIncrement>>)
----

through

[subs=normal]
----
<<SyntaxOutputStartLine,OutputStartLine>> + ((n + 1) * <<SyntaxOutputLineIncrement,OutputLineIncrement>>) - 1
----

If absent, <<SyntaxRepeatCount,RepeatCount>> and
<<SyntaxOutputLineIncrement,OutputLineIncrement>> default to one. If absent,
<<SyntaxLineFileID,LineFileID>> defaults to the most recent value (initially
zero).

The first line of a file is line one. <<SyntaxRepeatCount,RepeatCount>> is
greater than or equal to one. Each <<SyntaxLineFileID,LineFileID>> must be a
file ID present in the <<FileSection,FileSection>>.
<<SyntaxInputStartLine,InputStartLine>> is greater than or equal to one.
<<SyntaxOutputStartLine,OutputStartLine>> is greater than or equal to one.
<<SyntaxOutputLineIncrement,OutputLineIncrement>> is greater than or equal to
zero. A <<LineSection,LineSection>> may only occur after a
<<StratumSection,StratumSection>>.

For example:

 *L
 123:207
 130,3:210
 140:250,7
 160,3:300,2

Creates this mapping:

[width="100%",cols="3",options="header"]
|===
|Input Source
2+^|Output Source

|Line
|Begin Line
|End Line

|123
|207
|207

|130
|210
|210

|131
|211
|211

|132
|212
|212

|140
|250
|256

|160
|300
|301

|161
|302
|303

|162
|304
|305

|===

Note that multiple <<SyntaxLineInfo,LineInfo>> may map multiple input source
lines to a single output source line, when such a
<<SyntaxLineSection,LineSection>> is being used to map output source lines to
input source lines, a first matching <<SyntaxLineInfo,LineInfo>> rule applies.

Note also that multiple <<SyntaxLineInfo,LineInfo>> may map a single input
source line to a multiple, possibly disjoint, output source lines, when such a
<<SyntaxLineSection,LineSection>> is being used to map input source lines to
output source lines, a first matching <<SyntaxLineInfo,LineInfo>> rule again
applies.

=== VendorSection

The vendor section is for vendor specific information.

The format is `*V` on the first line to mark the section. The second line is the
vendor ID which is formed by the same rules by which unique package names are
formed in the Java language specification, third edition
https://docs.oracle.com/javase/specs/jls/se6/html/packages.html#7.7[(§7.7) Unique Package Names].

=== EndSection

The end section marks the end of an SMAP, it consists simply of a `*E` marker.
The end section must be the last line of an SMAP.

=== Embedded Source Maps

The <<SyntaxOpenEmbeddedSection,OpenEmbeddedSection>> marks the beginning and
<<SyntaxCloseEmbeddedSection,CloseEmbeddedSection>> the end of a set of
<<SyntaxEmbeddedSourceMaps,EmbeddedSourceMaps>>. These SMAPs correspond to the
input source for a language-processor. The stratum of the language-processor is
indicated on both sections. These sections must not occur in a
<<SMAP Resolution,resolved>> SMAP.

The format is the `*O` marker and the name of the output stratum on the first
line. This is followed by the set of embedded SMAPs. The embedded SMAPs are
included "whole" - from the `SMAP` to the <<EndSection,`EndSection`>> `*E`
marker - inclusive. Finally, the `*C` marker and the name of the output stratum
on the last line terminates the embedded SMAPs.

=== SMAP Syntax

[listing,subs="+macros"]
----
SMAP:
            Header { Section } EndSection

Header:
            ID OutputFileName DefaultStratumId

ID:
            SMAP CR

OutputFileName:
            NONASTERISKSTRING CR

DefaultStratumId:
            NONASTERISKSTRING CR

Section:
            StratumSection
            FileSection
            LineSection
            EmbeddedSourceMaps
            VendorSection
            FutureSection
[[SyntaxEmbeddedSourceMaps]]
EmbeddedSourceMaps:
            OpenEmbeddedSection { SMAP } CloseEmbeddedSection
[[SyntaxOpenEmbeddedSection]]
OpenEmbeddedSection:
            *O StratumID CR
[[SyntaxCloseEmbeddedSection]]
CloseEmbeddedSection:
            *C StratumID CR

StratumSection:
            *S StratumID CR

StratumID:
            NONASTERISKSTRING
[[SyntaxLineSection]]
LineSection:
            *L CR { LineInfo }
[[SyntaxLineInfo]]
LineInfo:
            InputLineInfo : OutputLineInfo CR

InputLineInfo:
            InputStartLine , RepeatCount
            InputStartLine

OutputLineInfo:
            OutputStartLine , OutputLineIncrement
            OutputStartLine
[[SyntaxInputStartLine]]
InputStartLine:
            NUMBER
            NUMBER # LineFileID
[[SyntaxLineFileID]]
LineFileID:
            FileID
[[SyntaxRepeatCount]]
RepeatCount:
            NUMBER
[[SyntaxOutputStartLine]]
OutputStartLine:
            NUMBER
[[SyntaxOutputLineIncrement]]
OutputLineIncrement:
            NUMBER

FileSection:
            *F CR { FileInfo }
[[SyntaxFileInfo]]
FileInfo:
            FileID FileName CR
            + FileID FileName CR AbsoluteFileName CR

FileID:
            NUMBER

FileName:
            NONASTERISKSTRING

AbsoluteFileName:
            NONASTERISKSTRING

VendorSection:
            *V CR VENDORID CR { VendorInfo }

VendorInfo:
            NONASTERISKSTRING CR

FutureSection:
            * OTHERCHAR CR { FutureInfo }

FutureInfo:
            NONASTERISKSTRING CR

EndSection:
            *E CR
----

Where `{x}` denotes zero or more occurrences of `x`. And where the terminals are
defined as follows (whitespace is a sequence of zero or more spaces or tabs):

[width="100%",cols="25%,75%",options="header"]
|===

2+^|Terminals

|`NONASTERISKSTRING`
|Any sequence of characters (excluding the terminal carriage-return or new-line)
which does not start with "*". Leading whitespace is ignored.

|`NUMBER`
|Non negative decimal integer. The number is terminated by the first non-digit
character. Leading and trailing whitespace is ignored.

|`CR`
|a line terminator: carriage-return, carriage-return followed by new-line or
new-line.

|`OTHERCHAR`
|Any character (other than carriage-return, new-line, space or tab) not already
used as a section header (not S,F,L,V,O,C or E).

|`VENDORID`
|A sequence of characters that identifies a vendor. The name is formed by the
same rules that unique package names are formed in the Java language
specification. Leading and trailing whitespace is ignored. The terminal
carriage-return or new-line is excluded.

|===


== SMAP Resolution

Before the SMAP in a SMAP-file can be installed into the SourceDebugExtension
attribute it must be resolved into an SMAP with no embedded SMAPs and with
final-source as the output source. A set of
<<SyntaxEmbeddedSourceMaps,embedded SMAPs>> is specific to a stratum and is
resolved in the context of the matching StratumSection in the outer SMAP. The
resolved SMAP includes StratumSections computed from each set of embedded SMAPs
as well as the unchanged StratumSections of the outer SMAP. If embedded SMAPs
are nested, the inner-most is resolved first.

The structure of an SMAP with embedded SMAPs is as follows:

[subs="normal,-replacements"]
----
SMAP
...
*O _B_
SMAP
...
*S _A_
...
*E
*C _B_
...
*S _B_
...
*E
----

The structure is a set of embedded SMAPs (for a stratum, here named _B_), an
outer StratumSection (for _B_), and an embedded SMAP with a StratumSection (for
a stratum, here named _A_). Note that: there may be many sets of embedded SMAPs,
many embedded SMAPs within the set of embedded SMAPs, and many StratumSections
within an SMAP. A StratumSection maps source information from its stratum to an
output stratum. Thus, the embedded StratumSection maps stratum _A_ to stratum
_B_. We know it is mapped to stratum _B_ because the set of embedded SMAPs for
stratum _B_ corresponds to the input for the language-processor for _B_. The
outer StratumSection maps stratum _B_ to its output stratum (let's call this
stratum _C_), if the shown SMAP is the outer-most SMAP then stratum _C_ is the
final-source stratum. The purpose of resolution is to create a non-embedded 
StratumSection for _A_ which maps to _C_ (all StratumSections within an SMAP
must map to the same output stratum, in a resolved outer-most SMAP all
StratumSections will map to the final-source stratum). This is done by
composing the mapping in the embedded StratumSection (from _A_ to _B_) with the
mapping in the outer StratumSection (from _B_ to _C_). Since there may be many
embedded StratumSections for _A_, these sections must be merged.

A StratumSection is computed for each stratum present in the embedded SMAPs. The
computed StratumSection is the merge of each embedded StratumSection, for that
stratum. Line number information is composed with the line number information of
the outer StratumSection (note that the embedded StratumSections cannot be for
the same stratum as the outer StratumSection). Specifically, a computed
StratumSection consists of a merged <<FileSection>>, a composed <<LineSection>>,
and direct copies of any <<VendorSection>>s or unknown sections. The merged
FileSection includes each unique <<SyntaxFileInfo,FileInfo>>, with FileIDs
reassigned to be unique. The composition the LineSections is described in the
algorithm below.

=== LineInfo Composition Algorithm

The following pseudo-code sketches the algorithm for resolving LineInfo in
embedded SMAPs. LineInfo resolution is by composition - discussed above. An
embedded LineInfo which maps stratum _A_ to stratum _B_ is composed with an
outer LineInfo which maps stratum _B_ to stratum _C_ to create a new resolved
LineInfo which maps stratum _A_ to stratum _C_.

The SMAPs and their components are marked by subscript:

* Embedded SMAP - level~E~
* Outer StratumSection - level~O~
* Resolved computed StratumSection - level~R~

The inputs and outputs of the algorithm are LineInfo tuples. Line information is
represented in this algorithm in its <<SyntaxLineInfo,LineInfo format>> which is
discussed in the <<LineSection>>. This algorithm is invoked for each LineInfo~E~
in each embedded SMAP.

[subs=normal]
----
ResolveLineInfo:
   InputStartLine~E~ #LineFileID~E~, RepeatCount~E~ : OutputStartLine~E~, OutputLineIncrement~E~
as follows:

if RepeatCount~E~ > 0 then {
  for each LineInfo~O~ in the stratum of the embedded SMAP:
    InputStartLine~O~ #LineFileID~O~, RepeatCount~O~: OutputStartLine~O~, OutputLineIncrement~O~
  which includes OutputStartLine~E~
    that is, InputStartLine~O~ + N == OutputStartLine~E~
      for some offset into the outer input range N where 0 <= N < RepeatCount~O~
  and for which LineFileID~O~ has a sourceName matching the embedded SMAP's OutputFileName {
    compute the number of outer mapping repeations which can be applied
    available := RepeatCount~O~ - N ;
    compute the number of embedded mapping repeations which can be applied
    completeCount := floor(available / OutputLineIncrement~E~) min RepeatCount~E~ ;
    if completeCount > 0 then {
      output resolved LineInfo
        InputStartLine~E~ # uniquify(LineFileID~E~), completeCount :
        (OutputStartLine~O~ + (N * OutputLineIncrement~O~)),
        (OutputLineIncrement~E~ * OutputLineIncrement~O~) ;
      ResolveLineInfo
        (InputStartLine~E~ + completeCount) #LineFileID~E~, (RepeatCount~E~ - completeCount) :
        (OutputStartLine~E~ + completeCount * OutputLineIncrement~E~), OutputLineIncrement~E~ ;
    } else {
      output resolved LineInfo
        InputStartLine~E~ # uniquify(LineFileID~E~), 1 :
        (OutputStartLine~O~ + (N * OutputLineIncrement~O~)), available ;
      ResolveLineInfo
        InputStartLine~E~ #LineFileID~E~, 1 :
        (OutputStartLine~E~ + available), (OutputLineIncrement~E~ - available) ;
      ResolveLineInfo
        (InputStartLine~E~ + 1) #LineFileID~E~, (RepeatCount~E~ - 1):
        (OutputStartLine~E~ + OutputLineIncrement~E~), OutputLineIncrement~E~ ;
    }
  }
}
----

where _uniquify_ converts a LineFileID~E~ to a corresponding LineFileID~R~

=== Resolution Example

The following example demonstrates resolution with this algorithm. The
<<example,general example>> will provide context before walking through this
example. In this example, `Incl.bar` is included by `Hi.bar`, but each is the
result of a prior translation.

image:resolution.png[resolution diagram,pdfwidth=75%]         

If the unresolved SMAP (in `Hi.java.smap`) is as follows:

[cols="<.^20%,<.^20%"]
|===
|`SMAP +
Hi.java +
Java`
|_Outer Header_

|`*O Bar`
|_<<SyntaxOpenEmbeddedSection,OpenEmbeddedSection>>_

|`SMAP +
Hi.bar +
Java +
*S Foo +
*F +
1 Hi.foo +
*L +
1#1,5:1,2 +
*E`
|_Embedded SMAP (Hi.bar)_

|`SMAP +
Incl.bar +
Java +
*S Foo +
*F +
1 Incl.foo +
*L 1#1,2:1,2 +
*E`
|_Embedded SMAP (Incl.bar)_

|`*C Bar`
|_<<SyntaxCloseEmbeddedSection,CloseEmbeddedSection>>_

|`*S Bar +
*F +
1 Hi.bar +
2 Incl.bar +
*L +
1#1:1 +
1#2,4:2 +
3#1,8:6`
|_Outer StratumSection_

|`*E`
|_Final <<EndSection>>_
|===

The merged level~R~ FileSection is (in stratum `Foo`):

----
*F
1 Hi.foo
2 Incl.foo
----

The computation proceeds as follows:

[cols="11%,13%,13%,11%,11%,41%",options="header",]
|===
|LineInfo~E~
|LineInfo~E~ +
recursion 1
|LineInfo~E~ +
recursion 2
|matching +
outer +
LineInfo~O~
|resolved +
LineInfo~R~
|discussion

|1#1,5:1,2
|
|
|1#1:1
|1#1,1:1,1
|`ResolveLineInfo` is called for `1#1,5:1,2` (from the first embedded SMAP -
OutputFileName is `Hi.bar`). `1#1:1` is found as the outer StratumSection
LineInfo~O~ with `InputStartLineO` of 1 and `LineFileIDO` has a sourceName
matching `Hi.bar`. `N` is 0, and `completeCount` is 0, thus the _else_ 
branch is taken. `available` is 1 and thus output is `1#1,1:1,1`.

|
|1#1,1:2,1
|
|no match
|
|The remaining half of the initial LineInfo~E~ mapping then must be resolved
recursively, but there is no match and it is ignored.

|
|2#1,4:3,2
|
|3#1,8:6
|2#1,4:6,2
|The remaining mappings are also handled recursively. There is a matching
LineInfo~O~. `N` is 0, and `completeCount` is 4, thus the _if_ branch is taken.

|
|
|6#1,0:11,2
|n/a
|
|The recursive resolve descends deeper but does nothing since all of
RepeatCount~E~ has been mapped. The first LineInfo~E~ is now resolved. Since it
had only one LineInfo~E~ the first SMAP is also resolved.

|1#1,2:1,2
|
|
|1#2,4:2
|1#2,2:2,2
|Now for the second SMAP (OutputFileName is `Incl.bar`). FileID~E~ 1 in this
SMAP is `Incl.foo` which corresponds to the remapped FileID~R~ 2. So the
matching LineInfo~O~ is `1#2,4:2`. `N` is 0, and `completeCount` is 2, so the
_if_ branch is taken.

|
|3#1,0:5,2
|
|n/a
|
|The recursive resolve does nothing since all the maps have been handled.
Resolution is complete.
|===

The resultant resolved SMAP is:

----
SMAP
Hi.java
Java
*S Foo
*F
1 Hi.foo
2 Incl.foo
*L
1#1,1:1,1
2#1,4:6,2
1#2,2:2,2
*S Bar
*F
1 Hi.bar
2 Incl.bar
*L
1#1:1
1#2,4:2
3#1,8:6
*E
----

== JPDA Support

The https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/[Java Platform Debugger
Architecture] in the Java 1.4 release was extended in support of debugging other
languages. The new APIs and APIs with comments changed to include reference to
<<TermStratum,strata>> are listed below:

[cols="60%,40%",options="headers"]
|===

|New APIs
|APIs with Changed Comments

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jvmdi-spec.html[JVMDI]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jvmdi-spec.html#GetSourceDebugExtension[`GetSourceDebugExtension`]
|

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp-spec.html[JDWP] -
`ReferenceType` (2) Command Set

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp/jdwp-protocol.html#JDWP_ReferenceType_SourceDebugExtension[`SourceDebugExtension`] Command (12)
|

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp-spec.html[JDWP] -
`VirtualMachine` (1) Command Set

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp/jdwp-protocol.html#JDWP_VirtualMachine_SetDefaultStratum[`SetDefaultStratum`] Command (19)
|

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/index.html[JDI] -
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/VirtualMachine.html[`VirtualMachine`]
interface

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/VirtualMachine.html#setDefaultStratum(java.lang.String)[`void setDefaultStratum(String stratum)`]
|

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/VirtualMachine.html#getDefaultStratum()[`String getDefaultStratum()`]
|

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/index.html[JDI] -
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html[`ReferenceType`] interface

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#sourceNames(java.lang.String)[`String sourceNames(String stratum)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#sourceName()[`String sourceName()`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#sourcePaths(java.lang.String)[`String sourcePaths(String stratum)`]
|

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#allLineLocations(java.lang.String,%20java.lang.String)[`List allLineLocations(String stratum, String sourceName)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#allLineLocations()[`List allLineLocations()`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#locationsOfLine(java.lang.String,%20java.lang.String,%20int)[`List locationsOfLine(String stratum, String sourceName, int lineNumber)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#locationsOfLine(int)[`List locationsOfLine(int lineNumber)`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#availableStrata()[`List availableStrata()`]
|

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#defaultStratum()[`String defaultStratum()`]
|

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#sourceDebugExtension()[`String sourceDebugExtension()`]
|

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/index.html[JDI] -
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Method.html[`Method`] interface

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Method.html#allLineLocations(java.lang.String,%20java.lang.String)[`List allLineLocations(String stratum, String sourceName)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Method.html#allLineLocations()[`List allLineLocations()`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Method.html#locationsOfLine(java.lang.String,%20java.lang.String,%20int)[`List locationsOfLine(String stratum, String sourceName, int lineNumber)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Method.html#locationsOfLine(int)[`List locationsOfLine(int lineNumber)`]

2+^|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/index.html[JDI] -
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html[`Location`] interface

|
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html[class comment] (strata defined)

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#lineNumber(java.lang.String)[`int lineNumber(String stratum)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#lineNumber()[`int lineNumber()`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#sourceName(java.lang.String)[`String sourceName(String stratum)`]
|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#sourceName()[`String sourceName()`]

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#sourcePath(java.lang.String)[`String sourcePath(String stratum)`]
|

|https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/Location.html#sourcePath()[`String sourcePath()`]
|

|===

== SourceDebugExtension Support

Debugger applications frequently need debugging information about the source
that exceeds what is delivered by the existing Java^TM^ Virtual Machine class
file attributes (SourceFile, LineNumber, and LocalVariable). This is
particularly true for debugging the source of other languages. In a distributed
environment side files may not be accessible, the information must be directly
associated with the class.

The solution is the addition of a class file attribute which holds a string.
The string contains debugging information in a standardized format which allows
for evolution and vendor extension.

=== SourceDebugExtension Access

This string is made opaquely accessible at the three layers of the
https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/[Java Platform Debugger Architecture] (JPDA):

[cols="10%,90%"]
|===
|JVMDI |https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jvmdi-spec.html#GetSourceDebugExtension[`GetSourceDebugExtension(jclass clazz, char **sourceDebugExtensionPtr)`]
|JDWP |https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdwp/jdwp-protocol.html#JDWP_ReferenceType_SourceDebugExtension[`SourceDebugExtension`] Command (12) in the `ReferenceType` (2) Command Set
|JDI |https://docs.oracle.com/javase/1.5.0/docs/guide/jpda/jdi/com/sun/jdi/ReferenceType.html#sourceDebugExtension()[`String sourceDebugExtension()`] in the `ReferenceType` interface
|===

[#attribute]
=== SourceDebugExtension Class File Attribute

Java virtual machine class file attributes are described in
https://docs.oracle.com/javase/specs/jvms/se6/html/ClassFile.doc.html#43817[section 4.7]
of the https://docs.oracle.com/javase/specs/jvms/se6/html/VMSpecTOC.doc.html[The Java Virtual Machine Specification].
The definition of the added attribute is in the context of The Java Virtual
Machine Specification:

[width="100%",cols="100%"]
|===
a|
The `SourceDebugExtension` attribute is an optional attribute in the
`attributes` table of the `ClassFile` structure. There can be no more than one
`SourceDebugExtension` attribute in the `attributes` table of a given `ClassFile`
structure.

The `SourceDebugExtension` attribute has the following format:

 SourceDebugExtension_attribute {
    u2 attribute_name_index;
    u4 attribute_length;
    u1 debug_extension[attribute_length];
 }

The items of the `SourceDebugExtension_attribute` structure are as follows:

`attribute_name_index`::
  The value of the `attribute_name_index` item must be a valid index into the
  `constant_pool` table. The `constant_pool` entry at that index must be a
  `CONSTANT_Utf8_info` structure representing the string
  `"SourceDebugExtension"`.
`attribute_length`::
  The value of the `attribute_length` item indicates the length of the
  attribute, excluding the initial six bytes. The value of the
  `attribute_length` item is thus the number of bytes in the `debug_extension[]`
  item.
`debug_extension[]`::
  The `debug_extension` array holds a string, which must be in UTF-8 format.
  There is no terminating zero byte. +
  The string in the `debug_extension` item will be interpreted as extended
  debugging information. The content of this string has no semantic effect on
  the Java Virtual Machine.

|===

== Example

The example below shows how the process described above would apply to a tiny
JSP program.

=== Input Source

The input consists of two JSP files, the first is `Hello.jsp`:

[cols="10%,90%",]
|===
|  1   |` <HTML>`
|  2   |` <HEAD>`
|  3   |` <TITLE>Hello Example</TITLE>`
|  4   |` </HEAD>`
|  5   |` <BODY>`
|  6   |` <%@ include file="greeting.jsp" %>`
|  7   |` </BODY>`
|  8   |` </HTML>`
|===

The second JSP file is the included file `greeting.jsp`:

[cols="10%,90%"]
|===
|  1   |` Hello There!<P>`
|  2   |` Goodbye on <%= new Date() %>`
|===

=== Language Processor

When a JSP compiler (the <<TermLanguageProcessor,language-processor>>) compiles
these files it will produce two outputs - a Java programming language source
file and a <<TermSourceMapFile,SMAP-file>>. The generated Java programming
language source file is `HelloServlet.java`:

[cols=">10%,90%"]
|===
|1 |`import jakarta.servlet.*;`
|2 |`import jakarta.servlet.http.*;`
|3 |
|4 |`public class HelloServlet extends HttpServlet {`
|5 |`  public void doGet(HttpServletRequest request,`
|6 |`      HttpServletResponse response)`
|7 |`      throws ServletException, IOException {`
|8 |`    response.setContentType("text/html");`
|9 |`    PrintWriter out = response.getWriter();`
|10|`    // Hello.jsp:1`
|11|`    out.println("<HTML>");`
|12|`    // Hello.jsp:2`
|13|`    out.println("<HEAD>");`
|14|`    // Hello.jsp:3`
|15|`    out.println("<TITLE>Hello Example</TITLE>");`
|16|`    // Hello.jsp:4`
|17|`    out.println("</HEAD>");`
|18|`    // Hello.jsp:5`
|19|`    out.println("<BODY>");`
|20|`    // greeting.jsp:1`
|21|`    out.println("Hello There!<P>");`
|22|`    // greeting.jsp:2`
|23|`    out.println("Goodbye on " + new Date() );`
|24|`    // Hello.jsp:7`
|25|`    out.println("</BODY>");`
|26|`    // Hello.jsp:8`
|27|`    out.println("</HTML>");`
|28|`  }`
|29|`}`
|===

The generated <<TermSourceMapFile,SMAP-file>> is `HelloServlet.java.smap`:

----
SMAP
HelloServlet.java
JSP
*S JSP
*F
1 Hello.jsp
2 greeting.jsp
*L
1#1,5:10,2
1#2,2:20,2
7#1,2:24,2
*E
----

A couple things are interesting to note about this SMAP -- the user has chosen
to make JSP the default stratum (perhaps by a command line option) and even
though there are ten lines of input source and 29 lines of generated source,
only three LineInfo lines describe the transformation: the first and last are
for the lines before and after the include (respectively) and the middle is for
the included file `greeting.jsp`.

The three LineInfo lines describe these mappings:

----
1#1,5:10,2
     Hello.jsp:    line 1 ->  HelloServlet.java: lines 10, 11
                   line 2 ->                     lines 12, 13
                   line 3 ->                     lines 14, 15
                   line 4 ->                     lines 16, 17
                   line 5 ->                     lines 18, 19

1#2,2:20,2
     greeting.jsp: line 1 ->  HelloServlet.java: lines 20, 21
                   line 2 ->                     lines 22, 23

7#1,2:24,2
      Hello.jsp:   line 7 ->  HelloServlet.java: lines 24, 25
                   line 8 ->                     lines 26, 27
----

=== Post Processor

Next `HelloServlet.java` is compiled by a Java programming language compiler
(for example `javac`) producing the class file `HelloServlet.class`. Then the
<<TermPostProcessor,post-processor>> is run. It takes `HelloServlet.class` and
`HelloServlet.java.smap` as input. It creates a
<<TermSourceDebugExtension,SourceDebugExtension>> attribute whose content is the
SMAP in `HelloServlet.java.smap` and rewrites `HelloServlet.class` with this
attribute.

=== Debugging

Now the program is run under the control of a debugger (which is a client of
<<TermJDI,JDI>>). Let's say we are stepping through this code and the debugger
has just received a <<TermJDI,JDI>> `StepEvent` for the line that is just about
to output `<BODY>`. The debugger's code might look like this (the `StepEvent`
is in the variable `stepEvent`):

----
Location location = stepEvent.location();
String sourceName = location.sourceName("Java");
int lineNumber = location.lineNumber("Java");
displaySource(sourceName, lineNumber);
----

where `displaySource` is a debugger routine that displays a source location.
Because the `Java` stratum has been specified `sourceName` would be
`HelloServlet.java`, the `lineNumber` would be `19` and the displayed line would
be:

----
out.println("<BODY>");
----

However, if `sourceName` and `lineNumber` were derived as follows:

----
String sourceName = location.sourceName("JSP");
int lineNumber = location.lineNumber("JSP");
----

Since the `JSP` stratum has been specified, `sourceName` would be `Hello.jsp`,
the `lineNumber` would be `5` and the displayed line would be:

----
<BODY>
----

This occurs because the <<TermSourceDebugExtension,SourceDebugExtension>>
attribute was stored when the VM read `HelloServlet.class` and it was retrieved
with the SourceDebugExtension JDWP command which in turn caused the JVMDI
function call GetSourceDebugExtension. The SMAP in the SourceDebugExtension was
parsed which provided the above transformation of source location. Specifically,
the line:

----
1#1,5:10,2
----

is the basis of this transformation - which refers to FileId #1

----
1 Hello.jsp
----

and whence the sourceName information. Since the default stratum specified in
the SMAP is `JSP`, the code:

----
String sourceName = location.sourceName();
int lineNumber = location.lineNumber();
----

would have the same effect. Since this is the form code would have taken before
these extensions were introduced, existing debuggers can be utilized if they are
run under the new implementation of <<TermJDI,JDI>>.