SQL editor: schema browser, Open in SQL Editor, cancel, date functions (2/4)

.github/workflows/main.yml

@@ -19,7 +19,7 @@ jobs:
   build-test:
     strategy:
       matrix:
-        java: [ 17 ]
+        java: [ 21 ]
         os: [ ubuntu-latest, windows-latest ]
     name: 'Test, ${{ matrix.os }}, java ${{ matrix.java }}'
     runs-on: ${{ matrix.os }}

CALCITE-1.42-QUERIES.md

@@ -0,0 +1,173 @@
+# What Calcite 1.42 adds to heap queries
+
+The plugin used to run on Calcite **1.36** in practice. That was the version in the target platform; see
+MIGRATION-NOTES §4.5. It now runs on **1.42**. This note covers what the six releases in between (1.37 to 1.42)
+add for SQL against a heap dump. It also covers what does **not** work yet with the plugin's configuration, and why.
+
+**Everything here was checked.** Every query below was run against the test dump
+(`MatCalciteTest/dumps/mvn1m_jdk18.hprof`) through the plugin's own connection. The check used a temporary probe
+test on 2026-10-15; the outputs shown are real. Release attribution comes from the
+[Calcite release history](https://calcite.apache.org/docs/history.html).
+
+The plugin's connection is `lex=JAVA`, `quoting=DOUBLE_QUOTE`, `conformance=LENIENT`, with no `fun` library
+(`CalciteDataSource.getConnection`). That setup decides most of what follows.
+
+---
+
+## 1. Works today, no change needed
+
+### Bitwise operators and functions (1.41, 1.38)
+
+`&`, `^` and `<<` (1.41), and `BITAND`, `BITOR`, `BITXOR`, `BITNOT`, `BITCOUNT` (1.38).
+Together with `getAddress()`, they make **object addresses** something you can analyse.
+
+Object alignment, grouping strings by the low 4 bits of their address:
+
+```sql
+select getAddress(this) & 15 a, count(*) c
+  from java.lang.String
+ group by getAddress(this) & 15
+```
+```
+a | c
+0 | 1597
+8 | 1659
+```
+
+The result: half the strings sit on 16-byte boundaries, half on 8 only. This is the kind of check behind
+compressed oops and object alignment questions.
+
+More examples:
+
+```sql
+select bitand(getAddress(this), 4095) page_offset from java.lang.String   -- position inside a 4 KiB page
+select bitcount(getAddress(this)) from java.lang.String                   -- bits set
+select getAddress(this) << 1, getAddress(this) ^ 1 from java.lang.String
+```
+
+### ASOF JOIN (1.38): "nearest object by address"
+
+An ASOF join matches each row with the **closest** row on the other side that satisfies an inequality. On a heap,
+that means memory neighbourhood: which object sits right before or after this one.
+
+For each `HashMap`, this finds the `String` at the nearest lower address:
+
+```sql
+select m.a map_address, s.a string_before
+  from (select 1 k, getAddress(this) a from java.util.HashMap) m
+  asof join (select 1 k, getAddress(this) a from java.lang.String) s
+    match_condition m.a > s.a
+    on m.k = s.k
+ order by m.a
+```
+```
+map_address  | string_before
+33283927176  | 33283901048
+33283929848  | 33283901048
+...
+```
+
+**Gotcha:** the `ON` part must compare plain **columns** with `=`. Expressions such as `getType(m.this) =
+getType(s.this)` or `1 = 1` are rejected ("ASOF JOIN condition must be a conjunction of equality comparisons of
+columns from both sides"). Compute everything in sub-selects first, as above. For a real per-group match, put the
+grouping key in both sub-selects, e.g. a class name column.
+
+### Window frames with `EXCLUDE CURRENT ROW` (1.38)
+
+This compares each object with **the others** of its kind, without self-joins:
+
+```sql
+select getType(this) t, retainedSize(this) r,
+       sum(retainedSize(this)) over (partition by getType(this)
+           rows between unbounded preceding and unbounded following
+           exclude current row) retained_by_the_others
+  from instanceof.java.util.AbstractMap
+ order by 1, 2
+```
+
+Use this to spot the one map that dominates its class: `r` much larger than `retained_by_the_others`.
+
+**Gotcha:** `avg(...)` over such a frame fails with `/ by zero` when a class has a single instance, since the
+frame is then empty. Use `sum(...) / nullif(count(...), 0)` instead.
+
+### Smaller additions
+
+- **`DUAL`** (1.39): `select 1 from dual` works, handy for testing expressions.
+- **Array comparison** (1.38): `array[1, 2] < array[1, 3]` returns `true`.
+
+Not new, but worth knowing: **`QUALIFY`** has worked since Calcite 1.32, and it is the shortest way to say "top
+N per class":
+
+```sql
+select getType(this) t, retainedSize(this) r
+  from instanceof.java.util.AbstractMap
+qualify row_number() over (partition by getType(this) order by retainedSize(this) desc) = 1
+```
+
+### Things you get without writing anything new
+
+- **Planner work.** 1.42 adds `LEFT_MARK` joins for hash and nested-loop joins, used for `IN` / `EXISTS`
+  sub-queries. 1.41 improves join reordering. These should help sub-query-heavy heap queries. *Not measured here.*
+
+---
+
+## 2. Works with a one-line configuration change: function libraries
+
+Calcite 1.37 to 1.42 added many functions to its **library** operator tables (PostgreSQL, Spark, Hive, Oracle…).
+They are only visible when the connection names them in the `fun` property. The plugin sets no `fun`, so today
+they are **not available**.
+
+With `fun=standard,postgresql,spark,hive` added to the connection properties in `CalciteDataSource`, all of these
+worked against the heap:
+
+| Function | Library | Example | Result on the test dump |
+|---|---|---|---|
+| `SPLIT_PART` (1.39) | PostgreSQL | `SPLIT_PART(getType(this), '.', 3)` | `HashMap` |
+| `STARTS_WITH` (1.37) | PostgreSQL | `where STARTS_WITH(getType(this), 'java.util')` | 177 maps |
+| `REGEXP_REPLACE` (1.38) | PostgreSQL | `REGEXP_REPLACE(getType(this), '^java\.util\.', '')` | `HashMap` |
+| `REGEXP_LIKE` (1.37) | several | `where REGEXP_LIKE(getType(this), 'Hash.*Map')` | 175 maps |
+| `SUBSTRING_INDEX` (1.38) | Spark | `SUBSTRING_INDEX(getType(this), '.', -1)` | simple class name |
+| `CONCAT_WS` (1.38) | Spark | `CONCAT_WS('|', getType(this), 'x')` | `java.util.HashMap|x` |
+| `REVERSE` (1.38) | Spark | `REVERSE(getType(this))` | `paMhsaH.litu.avaj` |
+| `INSTR` (1.39) | Hive | `INSTR(getType(this), 'Map')` | `15` |
+
+String functions on class names and on `toString(this)` are the main win: grouping by package, short class names
+and regex filters, without writing plugin functions.
+
+**Before turning it on:**
+
+- **Names must be UPPER CASE.** With `lex=JAVA`, function lookup is case-sensitive and library functions are
+  registered in upper case: `split_part(...)` fails, `SPLIT_PART(...)` works. Built-in standard functions
+  (`bitand`, `count`…) are matched case-insensitively and are not affected. This should be documented, or the
+  lexer settings revisited.
+- **Overloads can surprise.** `HEX(getAddress(this))` returned the hex of the *decimal digits*
+  (`3333323833393031303438`), not of the address. `STRING_TO_ARRAY(getType(this), '.')` (1.40) returned `[]`.
+  Check each function before relying on it.
+- **Libraries can change shared names.** Libraries redefine some common functions (`LOG`, `GREATEST`, `LEAST`,
+  `LENGTH`…). Turn on only the libraries you need, and run the test suite afterwards.
+
+---
+
+## 3. In the release notes, but not usable with the current setup
+
+| Feature | Release | What happened | What it would take |
+|---|---|---|---|
+| `SELECT * EXCLUDE (...)`, `* EXCEPT (...)`, `* REPLACE (... AS col)` | 1.42 | Parse error | The core parser never builds `SqlStarExclude`; only the **Babel** parser does. Needs the `calcite-babel` dependency (target platform + `Import-Package`) and `parserFactory=...SqlBabelParserImpl#FACTORY`. Not tried. |
+| `SELECT ... BY` (GROUP BY + ORDER BY shorthand) | 1.42 | Parse error | Same as above, presumably Babel. |
+| Lambdas `x -> ...` | 1.37 | Parse error | Lambdas are accepted only as function arguments. The only higher-order function (Spark `EXISTS`) collides with the `EXISTS` keyword in the core parser. |
+| Lateral column alias (`select f(x) r, r / 1024 kb`) | 1.40 | "Column 'r' not found" | Not enabled through the JDBC settings used here. |
+| `MEASURE` columns | 1.38 | `AGGREGATE` not found | Not usable through this connection. |
+| Table function in `FROM` without `TABLE()` | 1.37 | Parses, but `getMapEntries(m.this)` fails with "Table 'm' not found" | Correlated use still needs `lateral table(...)` or `cross apply table(...)`. |
+| `LEFT_SHIFT(x, n)` function | 1.41 | Not found, with or without libraries | Use the `<<` operator. |
+
+---
+
+## 4. Suggested follow-ups
+
+1. **Turn on a curated `fun` list** (start with `postgresql`) in `CalciteDataSource`. Mention upper-case names in
+   the SQL Reference tab, and add the functions to the schema browser's Functions group.
+2. **Evaluate the Babel parser** for `SELECT * EXCLUDE/REPLACE`. Heap tables have many columns, so
+   `select * exclude ("@class", "@classLoader")` would be a real ergonomic win. Check that it does not break the
+   plugin's own syntax: `this['field']`, `instanceof.` names, `cross apply`.
+3. **Add regression tests** for the ASOF join, `EXCLUDE CURRENT ROW` and bitwise queries above, so future Calcite
+   upgrades keep them working.

CHANGELOG.md

@@ -1,4 +1,120 @@
 ## [Unreleased]
+### Added
+- SQL function `getHexAddress(x)`: the object address as `0x...`, the form MAT command lines and labels use.
+- SQL date and time functions: `toDateTime(x[, zone])` shows heap date objects as e.g. `2026-10-01 11:11:11.224`
+  (java.util.Date, java.sql.Date/Time/Timestamp, Calendar, java.time Instant, LocalDate, LocalTime, LocalDateTime,
+  ZonedDateTime, OffsetDateTime, and numbers as epoch milliseconds). Also `toEpochMillis(x[, zone])`,
+  `millisToDateTime(long[, zone])`, `secondsToDateTime(long[, zone])` and `toDuration(x)`. Points in time default to
+  UTC, zoned values use their own zone, java.sql.Date/Time use the dumped JVM's zone; `'jvm'` selects that zone.
+- `instanceof."java.lang.ref.Reference"`: the instanceof schema takes a class's full name as one quoted
+  identifier, like the root schema does, so package parts that are SQL words (`ref`, `user`) need no quoting.
+- SQL editor: right-click menu with Run, Explain, Undo/Redo, Cut/Copy/Paste, Delete, Select All and
+  Toggle Comment.
+- SQL editor: "SQL Reference" toolbar button that opens tables, virtual columns and functions in a separate
+  tab, so the current result stays open. The editor now starts with a short hint and a sample query instead of the 30-line help comment.
+- SQL editor syntax colors are theme colors (Preferences > Appearance > Colors and Fonts > SQL (Calcite)
+  editor), with defaults for the dark theme.
+- SQL editor keys (F5 / Ctrl+Enter, F10, Ctrl+Space, Ctrl+/) are now workbench commands in their own key
+  binding context. MAT binds F5, Ctrl+Enter and Ctrl+Space to its OQL commands, and the workbench consumed
+  those keys before the SQL editor could see them.
+- SQL editor: schema browser beside the editor (toolbar toggle) listing classes by package, classes with
+  subclasses (instanceof), thread stacks and functions, with each class's columns and its this['@shallow'] /
+  this['@retained'] sizes. Double-click inserts the name at the cursor, quoted where SQL needs it; the filter box
+  lists matching tables.
+- "SQL > Open in SQL Editor" on the context menu of objects and classes in MAT's views (histogram, dominator
+  tree, object lists...). It adds a query for the selection to the last used SQL tab (a class row selects all
+  instances, objects are selected by address, one statement per class) and selects it without running it.
+- SQL editor: completion proposals pop up after typing '.' or '"', not only on Ctrl+Space.
+- SQL result rows show MAT's object icons (instance, class, array, class loader, GC root overlay).
+- Navigation history: re-opening a SQL result restores its query in the editor and runs it again, as the OQL
+  pane does.
+
+### Changed
+- **BREAKING**: Eclipse Memory Analyzer 1.17.0 or higher is now required (previously 1.14.0).
+  `org.eclipse.mat.api` 1.17.0 declares `Bundle-RequiredExecutionEnvironment: JavaSE-21`, so Java 21 is
+  now required. Users on older MAT must stay on plugin version 1.6.x.
+- **BREAKING**: The build target platform moved from Eclipse 2022-12 to Eclipse 4.40, matching the
+  Eclipse release that MAT 1.17.0 itself builds against.
+- Update Apache Calcite to 1.42.0 (requires Guava >= 33.4.8), with Avatica 1.29.0 instead of the 1.28.0 Calcite
+  asks for: Avatica before 1.29.0 has CVE-2026-70410 (unsafe reflection, CVSS 8.8). Avatica 1.29.0 brings
+  Jackson 2.22.2 in place of Calcite's 2.18.6.
+- MAT bundles are now resolved from the official MAT 1.17.0 update site
+  (`https://download.eclipse.org/mat/1.17.0/update-site/`) instead of an Eclipse simultaneous release,
+  so the MAT API version matches the MAT users run. The URL names the release: `latest` would move to
+  the next MAT release by itself.
+- SQL errors are shown without adding an entry to the navigation history, as the OQL pane does.
+- The SQL result area takes 4/5 of the pane height, as in the OQL pane.
+
+### Fixed
+- `getDominator(x)` failed with "Object -1 not found" for objects dominated by the virtual root (e.g. shared
+  objects); it now returns null.
+- `ORDER BY this` (and any `ORDER BY` over a heap object column) no longer fails with
+  `UnsupportedOperationException`, and sorts heap objects by address. Calcite 1.37+ compares `Map`-typed values
+  with `Functions.compareMaps()`, which walks `entrySet()`; `HeapReference` implements `Map` only to support
+  `this['field']` syntax and threw from `entrySet()`. It now returns one entry that orders like
+  `HeapReference.compareTo`, which orders by address. (An empty set, tried first, made every two heap objects
+  equal: `ORDER BY this` kept scan order, and merge joins on heap objects dropped matches, e.g. `count(cl.this)`
+  of `java.lang.Class left join instanceof.java.lang.ClassLoader` gave 469 instead of 985.) Both must give the
+  same order: Calcite sorts merge-join inputs through `entrySet()` but merges them with `compareTo`. Upstream's
+  `compareTo` ordered by label; by address is constant time, where a label was rebuilt on every comparison
+  (`ORDER BY this` over 4 million objects took about 2 minutes instead of 10 s). `ORDER BY toString(this)` sorts
+  by label. Sorting an object column in a result grid is by address too.
+- Added `httpclient5` and `httpcore5` (new Avatica dependencies since 1.28) and `org.jooq:joou-java-6` to the
+  target platform. Without these the plugin failed at runtime with
+  `NoClassDefFoundError: org/apache/hc/core5/util/Timeout` and `NoClassDefFoundError: org/joou/UInteger`.
+- Excluded `jakarta.transaction:jakarta.transaction-api` from `commons-dbcp2`. It declares an
+  unsatisfiable `Import-Package` (`javax.enterprise.context`) and blocked
+  `geronimo-jta_1.1_spec` from satisfying the `javax.transaction` packages Calcite and dbcp2 need.
+- Fixed a dependency drift bug where Calcite was declared twice with different versions:
+  `MatCalcitePlugin/pom.xml` said 1.41.0 while `MatCalciteTargetPlatform.target` said 1.36.0. The
+  plugin could compile against one Calcite and run against another. Both files are now kept in sync.
+- `testStackTraces` failed: MAT now renders a native frame with its source file,
+  `(Object.java(Native Method))` instead of `(Native Method)`. That text comes from MAT unchanged, so the
+  test ignores the source file part.
+- SQL editor: an error in a query run from a selection (F5 on selected text, which "Open in SQL Editor" sets up)
+  or from Explain was underlined at the wrong place: the position counted from the start of the editor and
+  ignored the `explain plan for` prefix. A position outside the text threw instead, and the error was not shown.
+- SQL editor: an error underline stayed after the query was fixed or another one ran; each run now clears it.
+- SQL editor: Cancel in the progress view did not stop a running query, and closing the tab left it running. Both
+  now stop the heap scan.
+- SQL editor: a query that finished after its tab was closed failed on the disposed editor. Closing the tab while
+  the editor had focus could leave its key bindings active and the Cut/Copy/Paste handlers pointing at it.
+- "Open in SQL Editor" on a class row of a partial histogram (Show Retained Set, a dominator tree grouped by class)
+  selected every instance of the class, not just the objects of the row.
+- "Open in SQL Editor" added the query to the active editor, which with several dumps open is not necessarily
+  the one the objects belong to; it now finds the editor of their dump.
+- Schema browser filter: it sorted every table name of the dump on each keystroke (four per class, slow on dumps
+  with 100,000 classes). The sorted names are now kept per schema, and the search runs when typing pauses.
+- `toDateTime(x)` returned the stale time of a `java.util.Date` changed with a deprecated setter (`setHours`...)
+  and of a `Calendar` changed with `set(...)` since its time was computed; it now returns null for them.
+- `toDuration(x)` overflowed beyond about 292 years: `Duration.ofSeconds(Long.MAX_VALUE)`, a common "forever",
+  showed as `-0s`.
+- Identifiers are quoted by Calcite's own reserved words, not only SQL-92's: the schema browser
+  wrote `java.lang.ref` unquoted, which does not parse (`REF` is reserved).
+- The SQL reference used `instanceof.java.util.Map` as its example, which is always empty: heap
+  dumps record superclasses but not interfaces. It now uses `java.util.AbstractMap` and says why.
+- Running a query could scroll the SQL editor when it was scrolled to the end: JFace's
+  ToolBarManager.relayout packs the toolbar's parent, which briefly lays the editor out taller, and StyledText
+  scrolls to fill the extra space. The editor now restores its scroll position after such a transient resize.
+- Line numbers of the SQL editor use the editor's colors, so they are readable in the dark theme.
+- Backspace, Delete and the arrow keys stopped working in the query editor after a table result was shown, while
+  letters still typed, until the window was deactivated and activated again. Cause: MAT shows table results with a
+  "Retrieving view elements..." user job whose progress dialog is opened and closed without ever getting focus. SWT
+  on GTK still records it as the active shell and, once it is disposed, has no active shell at all, so it drops every
+  key press that is not committed text. On Wayland the window gets no new focus-in event to repair that. The SQL
+  pane now re-activates the workbench window in that case (only if the window was never deactivated), which also
+  fixes MAT's own OQL pane.
+- The SQL pane leaked its title image and syntax colors. The title image now comes from the shared JFace image
+  registry (the navigation history keeps showing it after the tab is closed), and the colors from the theme
+  registry.
+- Undo and Redo keys (Ctrl+Z, Ctrl+Y) did not work in the SQL editor; the right-click menu's Cut and Copy could be
+  disabled while text was selected.
+- Removed debug output printed to stdout on every result table refresh and explain plan.
+
+### Notes
+- Only the MAT units this plugin needs are listed in the target (api, report, ui, parser, hprof).
+  Pulling the full `org.eclipse.mat.feature.feature.group` would drag in `mat.chart`/`mat.chart.ui`
+  and therefore BIRT and Rhino, which this plugin does not use.
 
 ## [1.6.2] - 2023-11-17
 ### Fixed

MatCalciteFeature/feature.xml

@@ -2,7 +2,7 @@
 <feature
         id="MatCalciteFeature"
         label="Calcite plugin for MAT"
-        version="1.6.4.qualifier"
+        version="2.0.0.qualifier"
         provider-name="Vladimir Sitnikov">
 
     <description>This plugin for Eclipse Memory Analyzer allows to query heap dump via SQL</description>

MatCalciteFeature/pom.xml

@@ -4,14 +4,14 @@
     <modelVersion>4.0.0</modelVersion>
 
     <artifactId>MatCalciteFeature</artifactId>
-    <version>1.6.4-SNAPSHOT</version>
+    <version>2.0.0-SNAPSHOT</version>
     <packaging>eclipse-feature</packaging>
     <name>Calcite SQL feature</name>
 
     <parent>
         <groupId>com.github.vlsi.mat.calcite</groupId>
         <artifactId>parent</artifactId>
-        <version>1.6.4-SNAPSHOT</version>
+        <version>2.0.0-SNAPSHOT</version>
     </parent>
 
     <dependencies>