From 6475d2a9559942731a75a6b6d5e42dea0e14c1ae Mon Sep 17 00:00:00 2001 From: miguel nieto Date: Fri, 16 Oct 2026 18:44:55 -0400 Subject: [PATCH 1/2] Upgrade to MAT 1.17.0, Java 21, Eclipse 4.40 and Calcite 1.42.0 BREAKING: requires MAT 1.17.0+ and Java 21. org.eclipse.mat.api 1.17.0 declares Bundle-RequiredExecutionEnvironment: JavaSE-21, so the execution environment cannot go lower. Users on older MAT must stay on 1.6.x. Platform - Target platform moved from Eclipse 2022-12 to Eclipse 4.40, matching the Eclipse release MAT 1.17.0 builds against. - MAT bundles come from the MAT 1.17.0 update site rather than an Eclipse simultaneous release, so the MAT API version matches the MAT users run. Only the units this plugin needs are listed; the full org.eclipse.mat.feature.feature.group would drag in mat.chart and therefore BIRT and Rhino, which are unused. - Java 21: BREE, executionEnvironment, compiler source/target, CI matrix, README requirements. Calcite 1.36 -> 1.42.0 - Aligned Calcite/Guava versions between MatCalcitePlugin/pom.xml and MatCalciteTargetPlatform.target (Guava 33.7.2-jre in both, after Renovate updated the pom only). They had drifted apart (1.41.0 vs 1.36.0) because Renovate only updates poms, not Tycho target definitions. - Avatica 1.29.0 instead of the 1.28.0 Calcite 1.42.0 asks for: Avatica before 1.29.0 has CVE-2026-70410 (unsafe reflection in plugin instantiation, CVSS 8.8). Declared before calcite-core so Maven compiles against the Jackson 2.22.2 it brings, which the target ships. - Added httpclient5/httpcore5 (new Avatica 1.28 dependencies) and org.jooq:joou-java-6 to the target platform and to Import-Package. Without them: NoClassDefFoundError for org/apache/hc/core5/util/Timeout and org/joou/UInteger. - Excluded jakarta.transaction:jakarta.transaction-api from commons-dbcp2: it has an unsatisfiable Import-Package (javax.enterprise.context) and blocked geronimo-jta_1.1_spec from supplying javax.transaction. ORDER BY and joins on heap objects - Calcite 1.37+ sorts Map-typed values with Functions.compareMaps(), which calls entrySet(); HeapReference is a Map only for this['field'] and threw from entrySet(), so every ORDER BY over a heap object failed. entrySet() now returns one entry that orders like compareTo. - Calcite sorts merge-join inputs through entrySet() but merges them with compareTo, so both must give the same order, or matches are dropped (the class/class loader join counted 469 instead of 985). Both now order by address, which is also constant time: ordering by label rebuilt the label on every comparison (ORDER BY this on 4M objects: about 2 min vs 10 s). ORDER BY toString(this) gives label order. SQL functions - getHexAddress(x): the object address as 0x..., as MAT shows it. - getDominator(x) returned an error for objects dominated by the virtual root; it now returns null. Tests - testStackTraces tolerates MAT's own stack frame formatting, which changed in MAT commit 0c72944e ("Bug 581644"): "(Native Method)" became "(Object.java(Native Method))". - New: ORDER BY this in both directions, a merge join over a strict subset, the class/class loader join phrased three ways (985 each). Co-Authored-By: Claude Opus 5.5 --- .github/workflows/main.yml | 2 +- CHANGELOG.md | 48 +++++++++++ MatCalciteFeature/feature.xml | 2 +- MatCalciteFeature/pom.xml | 4 +- MatCalcitePlugin/META-INF/MANIFEST.MF | 17 ++-- MatCalcitePlugin/pom.xml | 31 ++++++- .../vlsi/mat/calcite/HeapReference.java | 52 +++++++++-- .../mat/calcite/functions/HeapFunctions.java | 19 +++- MatCalciteRepository/pom.xml | 4 +- .../MatCalciteTargetPlatform.target | 86 +++++++++++++++++-- MatCalciteTargetPlatform/pom.xml | 4 +- MatCalciteTest/META-INF/MANIFEST.MF | 10 +-- MatCalciteTest/pom.xml | 2 +- .../tests/calcite/AbstractQueriesTests.java | 42 +++++++++ .../mat/tests/calcite/BasicQueriesTests.java | 52 ++++++++++- README.md | 8 +- pom.xml | 19 ++-- 17 files changed, 355 insertions(+), 47 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 8318827..072106d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -19,7 +19,7 @@ jobs: build-test: strategy: matrix: - java: [ 17 ] + java: [ 21 ] os: [ ubuntu-latest, windows-latest ] name: 'Test, ${{ matrix.os }}, java ${{ matrix.java }}' runs-on: ${{ matrix.os }} diff --git a/CHANGELOG.md b/CHANGELOG.md index b3e973e..4a489d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,52 @@ ## [Unreleased] +### Added +- SQL function `getHexAddress(x)`: the object address as `0x...`, the form MAT command lines and labels use. + +### Changed +- **BREAKING**: Eclipse Memory Analyzer 1.17.0 or higher is now required (previously 1.14.0). + `org.eclipse.mat.api` 1.17.0 declares `Bundle-RequiredExecutionEnvironment: JavaSE-21`, so Java 21 is + now required. Users on older MAT must stay on plugin version 1.6.x. +- **BREAKING**: The build target platform moved from Eclipse 2022-12 to Eclipse 4.40, matching the + Eclipse release that MAT 1.17.0 itself builds against. +- Update Apache Calcite to 1.42.0 (requires Guava >= 33.4.8), with Avatica 1.29.0 instead of the 1.28.0 Calcite + asks for: Avatica before 1.29.0 has CVE-2026-70410 (unsafe reflection, CVSS 8.8). Avatica 1.29.0 brings + Jackson 2.22.2 in place of Calcite's 2.18.6. +- MAT bundles are now resolved from the official MAT 1.17.0 update site + (`https://download.eclipse.org/mat/1.17.0/update-site/`) instead of an Eclipse simultaneous release, + so the MAT API version matches the MAT users run. The URL names the release: `latest` would move to + the next MAT release by itself. + +### Fixed +- `getDominator(x)` failed with "Object -1 not found" for objects dominated by the virtual root (e.g. shared + objects); it now returns null. +- `ORDER BY this` (and any `ORDER BY` over a heap object column) no longer fails with + `UnsupportedOperationException`, and sorts heap objects by address. Calcite 1.37+ compares `Map`-typed values + with `Functions.compareMaps()`, which walks `entrySet()`; `HeapReference` implements `Map` only to support + `this['field']` syntax and threw from `entrySet()`. It now returns one entry that orders like + `HeapReference.compareTo`, which orders by address. (An empty set, tried first, made every two heap objects + equal: `ORDER BY this` kept scan order, and merge joins on heap objects dropped matches, e.g. `count(cl.this)` + of `java.lang.Class left join instanceof.java.lang.ClassLoader` gave 469 instead of 985.) Both must give the + same order: Calcite sorts merge-join inputs through `entrySet()` but merges them with `compareTo`. Upstream's + `compareTo` ordered by label; by address is constant time, where a label was rebuilt on every comparison + (`ORDER BY this` over 4 million objects took about 2 minutes instead of 10 s). `ORDER BY toString(this)` sorts + by label. Sorting an object column in a result grid is by address too. +- Added `httpclient5` and `httpcore5` (new Avatica dependencies since 1.28) and `org.jooq:joou-java-6` to the + target platform. Without these the plugin failed at runtime with + `NoClassDefFoundError: org/apache/hc/core5/util/Timeout` and `NoClassDefFoundError: org/joou/UInteger`. +- Excluded `jakarta.transaction:jakarta.transaction-api` from `commons-dbcp2`. It declares an + unsatisfiable `Import-Package` (`javax.enterprise.context`) and blocked + `geronimo-jta_1.1_spec` from satisfying the `javax.transaction` packages Calcite and dbcp2 need. +- Fixed a dependency drift bug where Calcite was declared twice with different versions: + `MatCalcitePlugin/pom.xml` said 1.41.0 while `MatCalciteTargetPlatform.target` said 1.36.0. The + plugin could compile against one Calcite and run against another. Both files are now kept in sync. +- `testStackTraces` failed: MAT now renders a native frame with its source file, + `(Object.java(Native Method))` instead of `(Native Method)`. That text comes from MAT unchanged, so the + test ignores the source file part. + +### Notes +- Only the MAT units this plugin needs are listed in the target (api, report, ui, parser, hprof). + Pulling the full `org.eclipse.mat.feature.feature.group` would drag in `mat.chart`/`mat.chart.ui` + and therefore BIRT and Rhino, which this plugin does not use. ## [1.6.2] - 2023-11-17 ### Fixed diff --git a/MatCalciteFeature/feature.xml b/MatCalciteFeature/feature.xml index dbf026d..3937d04 100644 --- a/MatCalciteFeature/feature.xml +++ b/MatCalciteFeature/feature.xml @@ -2,7 +2,7 @@ This plugin for Eclipse Memory Analyzer allows to query heap dump via SQL diff --git a/MatCalciteFeature/pom.xml b/MatCalciteFeature/pom.xml index d21dfea..77854a0 100644 --- a/MatCalciteFeature/pom.xml +++ b/MatCalciteFeature/pom.xml @@ -4,14 +4,14 @@ 4.0.0 MatCalciteFeature - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT eclipse-feature Calcite SQL feature com.github.vlsi.mat.calcite parent - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT diff --git a/MatCalcitePlugin/META-INF/MANIFEST.MF b/MatCalcitePlugin/META-INF/MANIFEST.MF index efaef41..d0e8612 100644 --- a/MatCalcitePlugin/META-INF/MANIFEST.MF +++ b/MatCalcitePlugin/META-INF/MANIFEST.MF @@ -2,11 +2,11 @@ Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: MatCalcitePlugin Bundle-SymbolicName: MatCalcitePlugin;singleton:=true -Bundle-Version: 1.6.4.qualifier +Bundle-Version: 2.0.0.qualifier Bundle-Vendor: Vladimir Sitnikov -Bundle-RequiredExecutionEnvironment: JavaSE-17 -Require-Bundle: org.eclipse.mat.api;bundle-version="1.13.0", - org.eclipse.mat.ui;bundle-version="1.13.0", +Bundle-RequiredExecutionEnvironment: JavaSE-21 +Require-Bundle: org.eclipse.mat.api;bundle-version="1.17.0", + org.eclipse.mat.ui;bundle-version="1.17.0", org.eclipse.ui;bundle-version="3.6.0", org.eclipse.core.runtime;bundle-version="3.7.0", org.eclipse.jface.text;bundle-version="3.7.2", @@ -49,4 +49,11 @@ Import-Package: com.fasterxml.jackson.core.util, org.apache.commons.text.similarity, org.codehaus.commons.compiler, org.codehaus.janino, - org.locationtech.jts.geom + org.locationtech.jts.geom, + org.apache.hc.core5.util, + org.apache.hc.core5.http, + org.apache.hc.core5.http.io, + org.apache.hc.core5.net, + org.apache.hc.client5.http, + org.apache.hc.client5.http.classic, + org.joou diff --git a/MatCalcitePlugin/pom.xml b/MatCalcitePlugin/pom.xml index 5e1f19d..dd69f1a 100644 --- a/MatCalcitePlugin/pom.xml +++ b/MatCalcitePlugin/pom.xml @@ -4,14 +4,14 @@ 4.0.0 MatCalcitePlugin - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT eclipse-plugin Calcite SQL plugin com.github.vlsi.mat.calcite parent - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT @@ -25,15 +25,40 @@ json-smart 2.6.0 + + + org.apache.calcite.avatica + avatica-core + 1.29.0 + org.apache.calcite calcite-core - 1.41.0 + 1.42.0 org.apache.commons commons-dbcp2 2.14.0 + + + + jakarta.transaction + jakarta.transaction-api + + com.google.guava diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/HeapReference.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/HeapReference.java index b2a9587..2d6100a 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/HeapReference.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/HeapReference.java @@ -5,7 +5,9 @@ import org.eclipse.mat.snapshot.model.IArray; import org.eclipse.mat.snapshot.model.IObject; +import java.util.AbstractMap; import java.util.Collection; +import java.util.Collections; import java.util.Map; import java.util.Set; @@ -52,13 +54,17 @@ public String toString() { return o.getDisplayName(); } + /** + * Orders heap objects by address. Calcite must see one order everywhere: its sorts compare heap objects through + * {@link #entrySet} (they are Maps), while its merge join compares the same keys with this method, and two + * different orders made the merge join skip matches. Address order is also cheap: Calcite builds a new + * HeapReference each time a comparison reads a row, so ordering by label rebuilt the label (a String's text) on + * every comparison, and ORDER BY this on 4 million objects took minutes. ORDER BY toString(this) gives label + * order, computing one label per row. + */ @Override - public int compareTo(HeapReference o) { - int cmp = this.toString().compareTo(o.toString()); - if (cmp != 0) { - return cmp; - } - return getIObject().getObjectId() - o.getIObject().getObjectId(); + public int compareTo(HeapReference other) { + return Long.compare(o.getObjectAddress(), other.o.getObjectAddress()); } @Override @@ -132,8 +138,40 @@ public Collection values() { throw new UnsupportedOperationException(); } + /** + * One entry whose value orders like {@link #compareTo}, for Calcite. HeapReference is not a real Map: it is a + * view of one heap object, a Map only so that this['field'] maps onto {@link #get}. But Calcite sorts Map values + * (ORDER BY this, MIN(this), the inputs of a merge join) with Functions.compareMaps, which compares the entries in + * turn. An empty set made every two objects equal: ORDER BY this did not sort, and merge joins dropped matches. + * The value must not be a Map itself, or Calcite would compare it as one again; a Comparable of a single class + * makes it call {@link SortKey#compareTo}. + */ @Override public Set entrySet() { - throw new UnsupportedOperationException(); + return Collections.singleton(new AbstractMap.SimpleImmutableEntry<>("@this", new SortKey(this))); + } + + /** The order of {@link HeapReference#compareTo}, as a value Calcite compares with compareTo. */ + private static final class SortKey implements Comparable { + private final HeapReference ref; + + SortKey(HeapReference ref) { + this.ref = ref; + } + + @Override + public int compareTo(SortKey other) { + return ref.compareTo(other.ref); + } + + @Override + public boolean equals(Object other) { + return other instanceof SortKey && ref.equals(((SortKey) other).ref); + } + + @Override + public int hashCode() { + return ref.hashCode(); + } } } diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/HeapFunctions.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/HeapFunctions.java index 1a7f177..b507629 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/HeapFunctions.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/HeapFunctions.java @@ -233,6 +233,19 @@ public static long toLong(String value) { return Long.decode(value); } + /** + * Address of the referenced object as hex, e.g. 0x7bfeb5848: the form MAT uses for objects in command lines and + * labels. + */ + @SuppressWarnings("unused") + public static String getHexAddress(Object r) { + HeapReference ref = ensureHeapReference(r); + if (ref == null) { + return null; + } + return "0x" + Long.toHexString(ref.getIObject().getObjectAddress()); + } + @SuppressWarnings("unused") public static Object getDominator(Object r) { HeapReference ref = ensureHeapReference(r); @@ -242,7 +255,11 @@ public static Object getDominator(Object r) { try { ISnapshot snapshot = ref.getIObject().getSnapshot(); - return HeapReference.valueOf(snapshot.getObject(snapshot.getImmediateDominatorId(ref.getIObject().getObjectId()))); + int dominator = snapshot.getImmediateDominatorId(ref.getIObject().getObjectId()); + if (dominator < 0) { + return null; // dominated by the virtual root, e.g. an object referenced from several GC roots + } + return HeapReference.valueOf(snapshot.getObject(dominator)); } catch (SnapshotException e) { throw new RuntimeException("Cannot obtain immediate dominator object for " + r, e); } diff --git a/MatCalciteRepository/pom.xml b/MatCalciteRepository/pom.xml index 2ada50b..2c475dc 100644 --- a/MatCalciteRepository/pom.xml +++ b/MatCalciteRepository/pom.xml @@ -4,14 +4,14 @@ 4.0.0 MatCalciteRepository - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT eclipse-repository Calcite SQL update site com.github.vlsi.mat.calcite parent - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT diff --git a/MatCalciteTargetPlatform/MatCalciteTargetPlatform.target b/MatCalciteTargetPlatform/MatCalciteTargetPlatform.target index 984fdb4..e0c8761 100644 --- a/MatCalciteTargetPlatform/MatCalciteTargetPlatform.target +++ b/MatCalciteTargetPlatform/MatCalciteTargetPlatform.target @@ -1,40 +1,112 @@ - - + + + - + + + + + + + + + + + net.minidev accessors-smart - 2.5.0 + 2.6.0 net.minidev json-smart - 2.5.0 + 2.6.0 org.apache.calcite calcite-core - 1.36.0 + 1.42.0 + + + + org.apache.calcite.avatica + avatica-core + 1.29.0 com.google.guava guava - 32.1.3-jre + 33.7.2-jre com.google.code.findbugs jsr305 3.0.2 + + + org.apache.geronimo.specs + geronimo-jta_1.1_spec + 1.1.1 + + + + org.apache.httpcomponents.client5 + httpclient5 + 5.5 + + + org.apache.httpcomponents.core5 + httpcore5 + 5.3.5 + + + + org.jooq + joou-java-6 + 0.9.5 + 4.0.0 MatCalciteTargetPlatform - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT eclipse-target-definition Eclipse target definition parent com.github.vlsi.mat.calcite - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT diff --git a/MatCalciteTest/META-INF/MANIFEST.MF b/MatCalciteTest/META-INF/MANIFEST.MF index acd1b6e..b1755dd 100644 --- a/MatCalciteTest/META-INF/MANIFEST.MF +++ b/MatCalciteTest/META-INF/MANIFEST.MF @@ -2,12 +2,12 @@ Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Name: MatCalciteTest Bundle-SymbolicName: MatCalciteTest -Bundle-Version: 1.6.4.qualifier +Bundle-Version: 2.0.0.qualifier Require-Bundle: org.junit, org.eclipse.core.runtime, - org.eclipse.mat.api;bundle-version="1.8.0", - org.eclipse.mat.parser;bundle-version="1.8.0", - org.eclipse.mat.ui;bundle-version="1.8.0", - org.eclipse.mat.hprof;bundle-version="1.8.0" + org.eclipse.mat.api;bundle-version="1.17.0", + org.eclipse.mat.parser;bundle-version="1.17.0", + org.eclipse.mat.ui;bundle-version="1.17.0", + org.eclipse.mat.hprof;bundle-version="1.17.0" Import-Package: com.github.vlsi.mat.calcite, com.google.common.base diff --git a/MatCalciteTest/pom.xml b/MatCalciteTest/pom.xml index 610274d..783a07d 100644 --- a/MatCalciteTest/pom.xml +++ b/MatCalciteTest/pom.xml @@ -10,7 +10,7 @@ com.github.vlsi.mat.calcite parent - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AbstractQueriesTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AbstractQueriesTests.java index e5d9198..336315f 100644 --- a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AbstractQueriesTests.java +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AbstractQueriesTests.java @@ -43,6 +43,48 @@ protected void returnsInOrder(String sql, String... expected) throws SQLExceptio Assert.assertArrayEquals(sql, nnl(expected), nnl(actuals)); } + /** + * Same as {@link #returnsInOrder(String, String...)}, but tolerant of MAT's own stack frame formatting. + * + * MAT renders the source location of a native method differently depending on its version. Before commit + * 0c72944e ("Bug 581644 - Allow stack frames as pseudo-objects for HPROF snapshots") it produced + * "(Native Method)"; from that commit on it also includes the source file, e.g. + * "(Object.java(Native Method))". + * + * That string never passes through plugin logic - SnapshotThreadStacksTable returns + * IStackFrame.getText() verbatim - so pinning the exact rendering only makes the test fail whenever MAT + * changes its own formatting, for reasons unrelated to what is actually being tested. This helper strips + * the optional source file prefix so the assertion still covers the stable, meaningful part: that + * native.ThreadStackFrames is queryable, sorts correctly, and reports the right thread, depth and method. + */ + protected void returnsInOrderIgnoringStackFrameSourceFile(String sql, String... expected) + throws SQLException { + String[] actuals; + try { + actuals = executeToCSV(sql).toArray(new String[0]); + } catch (SQLException e) { + e.printStackTrace(); // tycho-surefire-plugin forces trimStackTrace=true + throw e; + } + Assert.assertArrayEquals(sql, stripStackFrameSourceFile(nnl(expected)), + stripStackFrameSourceFile(nnl(actuals))); + } + + private static Object[] stripStackFrameSourceFile(Object[] lines) { + for (int i = 0; i < lines.length; i++) { + if (lines[i] instanceof String) { + // "(Object.java(Native Method))" -> "(Native Method)". + // The [^()] guard keeps ordinary "(ReferenceQueue.java:142)" line references untouched, because + // they are followed by ':' rather than by '('. Note the two trailing \\): one closes the inner + // "(Native Method)", the other closes the outer "(...java(Native Method))". Omitting the second + // leaves a dangling ")" in the result. + lines[i] = ((String) lines[i]).replaceAll( + "\\((?:[^()]+\\.(?:java|class))\\((Native Method|Compiled Code)\\)\\)", "($1)"); + } + } + return lines; + } + protected List executeToCSV(String sql) throws SQLException { List res = new ArrayList<>(); System.out.println("sql = " + sql); diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java index beb7fe5..bba2f97 100644 --- a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java @@ -60,7 +60,11 @@ public void testStackTraceCount() throws SQLException { @Test public void testStackTraces() throws SQLException { - returnsInOrder("select thread, depth, text from native.ThreadStackFrames order by 1, 2, 3 limit 2", + // Uses the stack-frame-tolerant variant: MAT's exact rendering of a native method's source location + // changed between MAT versions ("(Native Method)" vs "(Object.java(Native Method))"), and that text is + // produced by MAT, never by this plugin. See returnsInOrderIgnoringStackFrameSourceFile. + returnsInOrderIgnoringStackFrameSourceFile( + "select thread, depth, text from native.ThreadStackFrames order by 1, 2, 3 limit 2", "thread|depth|text", "Finalizer|0|at java.lang.Object.wait(J)V (Native Method)", "Finalizer|1|at java.lang.ref.ReferenceQueue.remove(J)Ljava/lang/ref/Reference; (ReferenceQueue.java:142)"); @@ -163,6 +167,19 @@ public void selectClassFields() throws SQLException { "loader>"); } + @Test + public void orderByThisSortsByAddress() throws SQLException { + // ORDER BY this used to keep the scan order whichever direction was asked + returnsInOrder("select getHexAddress(this) a from java.lang.String where toString(this) like 'java.util.Hash%'" + + " order by this", + "a", + "0x7bfe83a00", "0x7bfe92df0", "0x7bfeaaf10"); + returnsInOrder("select getHexAddress(this) a from java.lang.String where toString(this) like 'java.util.Hash%'" + + " order by this desc", + "a", + "0x7bfeaaf10", "0x7bfe92df0", "0x7bfe83a00"); + } + @Test public void groupByClassLoader() throws SQLException { returnsInOrder("select \"@classLoader\", count(*) classes" + @@ -185,14 +202,47 @@ public void joinClassClass() throws SQLException { "985|985"); } + @Test + public void mergeJoinOnHeapObjectsKeepsEveryMatch() throws SQLException { + // Calcite sorts both inputs of a merge join through HeapReference.entrySet() and merges them with + // HeapReference.compareTo: when the two orders differed, matches were skipped. The right side must be a strict + // subset of the left for that to show (identical sides merge in lockstep): every third string by address, + // 1064 of the 3256 strings, each of which must find itself + returnsInOrder("select count(*) n, count(b.this) matched from java.lang.String a" + + " left join (select this from java.lang.String where mod(getAddress(this) / 8, 3) = 0) b" + + " on (a.this = b.this)", + "n|matched", + "3256|1064"); + } + @Test public void joinClassClassLoader() throws SQLException { + // Every class has its loader among the ClassLoader rows: the bootstrap loader is MAT's pseudo-object + // java.lang.ClassLoader @ 0x0, which instanceof.java.lang.ClassLoader includes. So all 985 rows match. + // This read 469 while every two heap objects compared equal (HeapReference.entrySet() was empty), which + // made a sort-based join drop matches. returnsInOrder("select count(c.name) c, count(cl.this) cl\n" + " from java.lang.Class c\n" + " left join instanceof.java.lang.ClassLoader cl\n" + " on (c.\"@classLoader\" = cl.this)", "c|cl", "985|985"); + // The same join phrased differently must agree + returnsInOrder("select count(*) n\n" + + " from java.lang.Class c\n" + + " left join instanceof.java.lang.ClassLoader cl\n" + + " on (c.\"@classLoader\" = cl.this)\n" + + " where cl.this is not null", + "n", + "985"); + // Classes per loader, as counted one loader at a time by address: 748 + 217 + 16 + 4 = 985 + returnsInOrder("select getHexAddress(cl.this) loader, count(*) classes\n" + + " from java.lang.Class c\n" + + " join instanceof.java.lang.ClassLoader cl\n" + + " on (c.\"@classLoader\" = cl.this)\n" + + " group by cl.this order by 2 desc", + "loader|classes", + "0x0|748", "0x7bfe97510|217", "0x7bfe391f0|16", "0x7bfe394f0|4"); } @Test diff --git a/README.md b/README.md index edac6b8..d6d2115 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ See [Calcite SQL reference](https://calcite.apache.org/docs/reference.html) Installation ------------ -Requirements: Java 17+, Eclipse Memory Analyzer 0.14.0+ +Requirements: Java 21+, Eclipse Memory Analyzer 1.17.0+ (for older MAT versions, use plugin version 1.6.x) There are two update channels: * https://vlsi.github.io/mat-calcite-plugin-update-site/stable (stable releases) @@ -231,9 +231,9 @@ from Requirements ------------ -Java 17 as a build JDK. -The code can use all Java 17 features. -Eclipse Memory Analyzer 0.14.0 or higher +Java 21 as a build JDK. +The code can use all Java 21 features. +Eclipse Memory Analyzer 1.17.0 or higher Building -------- diff --git a/pom.xml b/pom.xml index 7a1504b..b1834c4 100644 --- a/pom.xml +++ b/pom.xml @@ -5,16 +5,20 @@ com.github.vlsi.mat.calcite parent - 1.6.4-SNAPSHOT + 2.0.0-SNAPSHOT pom MAT Calcite plugin parent 4.0.13 - https://download.eclipse.org/releases/2022-12 + + https://download.eclipse.org/eclipse/updates/4.40 + + https://download.eclipse.org/mat/1.17.0/update-site/ UTF-8 - 17 - 17 + + 21 + 21 @@ -31,6 +35,11 @@ p2 ${eclipse-repository} + + MAT + p2 + ${mat-repository} + @@ -118,7 +127,7 @@ ${tycho-version} wrapAsBundle - JavaSE-17 + JavaSE-21 com.github.vlsi.mat.calcite From e1d5ea1426c5d0934f41bf3cb1829452cdfd0d45 Mon Sep 17 00:00:00 2001 From: miguel nieto Date: Fri, 16 Oct 2026 18:46:20 -0400 Subject: [PATCH 2/2] SQL editor on par with MAT's panes: schema browser, Open in SQL Editor, date functions Editor - Right-click menu (Run, Explain, Undo/Redo, Cut/Copy/Paste, Delete, Select All, Toggle Comment); keys are workbench commands in their own context, so F5, Ctrl+Enter and Ctrl+Space no longer go to MAT's OQL commands. - Schema browser beside the editor: classes by package, instanceof, thread stacks and functions, with columns and sizes; double-click inserts the name, quoted where SQL needs it. The filter is debounced and uses cached sorted names. - "SQL Reference" tab; theme colors (with dark theme defaults); completion pops up after '.' and '"'; object icons in results; navigation history restores and reruns a query. - Error underline at the right place for a selection and for Explain, and cleared on the next run. Cancel in the progress view and closing the tab stop a running query (QueryCancellation, checked by the heap scan). - Fixes: keys lost after a table result on GTK/Wayland (no active shell), editor scroll jump on a transient resize, Undo/Redo keys, leaked title image and colors, closed-tab cleanup, debug output on stdout. Open in SQL Editor - On MAT's object menus: a query for the selected objects (by address, one statement per class) or for a class row (all instances, or the row's objects for a partial histogram), added to the SQL tab of the objects' own dump. SQL - Date and time functions: toDateTime, toEpochMillis, millisToDateTime, secondsToDateTime, toDuration. - instanceof."java.lang.ref.Reference" (full class name as one identifier); identifiers quoted by Calcite's reserved words, not only SQL-92's. - CALCITE-1.42-QUERIES.md: what Calcite 1.37-1.42 adds to heap queries. Co-Authored-By: Claude Opus 5.5 --- CALCITE-1.42-QUERIES.md | 173 ++++++ CHANGELOG.md | 68 +++ MatCalcitePlugin/plugin.xml | 143 +++++ MatCalcitePlugin/resources/css/dark.css | 8 + .../vlsi/mat/calcite/CalciteDataSource.java | 17 + .../com/github/vlsi/mat/calcite/Query.java | 5 +- .../vlsi/mat/calcite/QueryCancellation.java | 51 ++ .../github/vlsi/mat/calcite/RowSetTable.java | 24 +- .../vlsi/mat/calcite/SchemaCatalog.java | 121 ++++ .../vlsi/mat/calcite/SqlForObjects.java | 91 +++ .../vlsi/mat/calcite/SqlIdentifiers.java | 46 ++ .../github/vlsi/mat/calcite/SqlReference.java | 61 ++ .../vlsi/mat/calcite/action/CalciteJob.java | 46 +- .../calcite/action/ExecuteQueryAction.java | 20 +- .../mat/calcite/editor/ActiveShellFix.java | 75 +++ .../CalciteContentAssistantProcessor.java | 3 +- .../vlsi/mat/calcite/editor/CalcitePane.java | 522 +++++++++++++++--- .../mat/calcite/editor/CalciteScanner.java | 10 +- .../CalciteSourceViewerConfiguration.java | 5 +- .../mat/calcite/editor/CommentScanner.java | 7 +- .../calcite/editor/OpenInSqlEditorQuery.java | 98 ++++ .../mat/calcite/editor/SchemaBrowser.java | 383 +++++++++++++ .../vlsi/mat/calcite/editor/SqlColors.java | 29 + .../editor/SqlEditorCommandHandler.java | 37 ++ .../mat/calcite/editor/SqlReferencePane.java | 46 ++ .../mat/calcite/editor/StringScanner.java | 7 +- .../editor/TransientResizeScrollFix.java | 75 +++ .../mat/calcite/functions/DateFunctions.java | 431 +++++++++++++++ .../vlsi/mat/calcite/neo/PackageSchema.java | 5 + .../objects/InstanceIdsByClassTable.java | 38 +- .../vlsi/mat/tests/calcite/AllTests.java | 6 +- .../mat/tests/calcite/BasicQueriesTests.java | 42 ++ .../mat/tests/calcite/DateFunctionsTests.java | 58 ++ .../tests/calcite/QueryCancellationTests.java | 61 ++ .../mat/tests/calcite/RowSetTableTests.java | 32 ++ .../mat/tests/calcite/SqlForObjectsTests.java | 93 ++++ 36 files changed, 2799 insertions(+), 138 deletions(-) create mode 100644 CALCITE-1.42-QUERIES.md create mode 100644 MatCalcitePlugin/resources/css/dark.css create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/QueryCancellation.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SchemaCatalog.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlForObjects.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlIdentifiers.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlReference.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/ActiveShellFix.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/OpenInSqlEditorQuery.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SchemaBrowser.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlColors.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlEditorCommandHandler.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlReferencePane.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/TransientResizeScrollFix.java create mode 100644 MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/DateFunctions.java create mode 100644 MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/DateFunctionsTests.java create mode 100644 MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/QueryCancellationTests.java create mode 100644 MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/RowSetTableTests.java create mode 100644 MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/SqlForObjectsTests.java diff --git a/CALCITE-1.42-QUERIES.md b/CALCITE-1.42-QUERIES.md new file mode 100644 index 0000000..a93e214 --- /dev/null +++ b/CALCITE-1.42-QUERIES.md @@ -0,0 +1,173 @@ +# What Calcite 1.42 adds to heap queries + +The plugin used to run on Calcite **1.36** in practice. That was the version in the target platform; see +MIGRATION-NOTES §4.5. It now runs on **1.42**. This note covers what the six releases in between (1.37 to 1.42) +add for SQL against a heap dump. It also covers what does **not** work yet with the plugin's configuration, and why. + +**Everything here was checked.** Every query below was run against the test dump +(`MatCalciteTest/dumps/mvn1m_jdk18.hprof`) through the plugin's own connection. The check used a temporary probe +test on 2026-10-15; the outputs shown are real. Release attribution comes from the +[Calcite release history](https://calcite.apache.org/docs/history.html). + +The plugin's connection is `lex=JAVA`, `quoting=DOUBLE_QUOTE`, `conformance=LENIENT`, with no `fun` library +(`CalciteDataSource.getConnection`). That setup decides most of what follows. + +--- + +## 1. Works today, no change needed + +### Bitwise operators and functions (1.41, 1.38) + +`&`, `^` and `<<` (1.41), and `BITAND`, `BITOR`, `BITXOR`, `BITNOT`, `BITCOUNT` (1.38). +Together with `getAddress()`, they make **object addresses** something you can analyse. + +Object alignment, grouping strings by the low 4 bits of their address: + +```sql +select getAddress(this) & 15 a, count(*) c + from java.lang.String + group by getAddress(this) & 15 +``` +``` +a | c +0 | 1597 +8 | 1659 +``` + +The result: half the strings sit on 16-byte boundaries, half on 8 only. This is the kind of check behind +compressed oops and object alignment questions. + +More examples: + +```sql +select bitand(getAddress(this), 4095) page_offset from java.lang.String -- position inside a 4 KiB page +select bitcount(getAddress(this)) from java.lang.String -- bits set +select getAddress(this) << 1, getAddress(this) ^ 1 from java.lang.String +``` + +### ASOF JOIN (1.38): "nearest object by address" + +An ASOF join matches each row with the **closest** row on the other side that satisfies an inequality. On a heap, +that means memory neighbourhood: which object sits right before or after this one. + +For each `HashMap`, this finds the `String` at the nearest lower address: + +```sql +select m.a map_address, s.a string_before + from (select 1 k, getAddress(this) a from java.util.HashMap) m + asof join (select 1 k, getAddress(this) a from java.lang.String) s + match_condition m.a > s.a + on m.k = s.k + order by m.a +``` +``` +map_address | string_before +33283927176 | 33283901048 +33283929848 | 33283901048 +... +``` + +**Gotcha:** the `ON` part must compare plain **columns** with `=`. Expressions such as `getType(m.this) = +getType(s.this)` or `1 = 1` are rejected ("ASOF JOIN condition must be a conjunction of equality comparisons of +columns from both sides"). Compute everything in sub-selects first, as above. For a real per-group match, put the +grouping key in both sub-selects, e.g. a class name column. + +### Window frames with `EXCLUDE CURRENT ROW` (1.38) + +This compares each object with **the others** of its kind, without self-joins: + +```sql +select getType(this) t, retainedSize(this) r, + sum(retainedSize(this)) over (partition by getType(this) + rows between unbounded preceding and unbounded following + exclude current row) retained_by_the_others + from instanceof.java.util.AbstractMap + order by 1, 2 +``` + +Use this to spot the one map that dominates its class: `r` much larger than `retained_by_the_others`. + +**Gotcha:** `avg(...)` over such a frame fails with `/ by zero` when a class has a single instance, since the +frame is then empty. Use `sum(...) / nullif(count(...), 0)` instead. + +### Smaller additions + +- **`DUAL`** (1.39): `select 1 from dual` works, handy for testing expressions. +- **Array comparison** (1.38): `array[1, 2] < array[1, 3]` returns `true`. + +Not new, but worth knowing: **`QUALIFY`** has worked since Calcite 1.32, and it is the shortest way to say "top +N per class": + +```sql +select getType(this) t, retainedSize(this) r + from instanceof.java.util.AbstractMap +qualify row_number() over (partition by getType(this) order by retainedSize(this) desc) = 1 +``` + +### Things you get without writing anything new + +- **Planner work.** 1.42 adds `LEFT_MARK` joins for hash and nested-loop joins, used for `IN` / `EXISTS` + sub-queries. 1.41 improves join reordering. These should help sub-query-heavy heap queries. *Not measured here.* + +--- + +## 2. Works with a one-line configuration change: function libraries + +Calcite 1.37 to 1.42 added many functions to its **library** operator tables (PostgreSQL, Spark, Hive, Oracle…). +They are only visible when the connection names them in the `fun` property. The plugin sets no `fun`, so today +they are **not available**. + +With `fun=standard,postgresql,spark,hive` added to the connection properties in `CalciteDataSource`, all of these +worked against the heap: + +| Function | Library | Example | Result on the test dump | +|---|---|---|---| +| `SPLIT_PART` (1.39) | PostgreSQL | `SPLIT_PART(getType(this), '.', 3)` | `HashMap` | +| `STARTS_WITH` (1.37) | PostgreSQL | `where STARTS_WITH(getType(this), 'java.util')` | 177 maps | +| `REGEXP_REPLACE` (1.38) | PostgreSQL | `REGEXP_REPLACE(getType(this), '^java\.util\.', '')` | `HashMap` | +| `REGEXP_LIKE` (1.37) | several | `where REGEXP_LIKE(getType(this), 'Hash.*Map')` | 175 maps | +| `SUBSTRING_INDEX` (1.38) | Spark | `SUBSTRING_INDEX(getType(this), '.', -1)` | simple class name | +| `CONCAT_WS` (1.38) | Spark | `CONCAT_WS('|', getType(this), 'x')` | `java.util.HashMap|x` | +| `REVERSE` (1.38) | Spark | `REVERSE(getType(this))` | `paMhsaH.litu.avaj` | +| `INSTR` (1.39) | Hive | `INSTR(getType(this), 'Map')` | `15` | + +String functions on class names and on `toString(this)` are the main win: grouping by package, short class names +and regex filters, without writing plugin functions. + +**Before turning it on:** + +- **Names must be UPPER CASE.** With `lex=JAVA`, function lookup is case-sensitive and library functions are + registered in upper case: `split_part(...)` fails, `SPLIT_PART(...)` works. Built-in standard functions + (`bitand`, `count`…) are matched case-insensitively and are not affected. This should be documented, or the + lexer settings revisited. +- **Overloads can surprise.** `HEX(getAddress(this))` returned the hex of the *decimal digits* + (`3333323833393031303438`), not of the address. `STRING_TO_ARRAY(getType(this), '.')` (1.40) returned `[]`. + Check each function before relying on it. +- **Libraries can change shared names.** Libraries redefine some common functions (`LOG`, `GREATEST`, `LEAST`, + `LENGTH`…). Turn on only the libraries you need, and run the test suite afterwards. + +--- + +## 3. In the release notes, but not usable with the current setup + +| Feature | Release | What happened | What it would take | +|---|---|---|---| +| `SELECT * EXCLUDE (...)`, `* EXCEPT (...)`, `* REPLACE (... AS col)` | 1.42 | Parse error | The core parser never builds `SqlStarExclude`; only the **Babel** parser does. Needs the `calcite-babel` dependency (target platform + `Import-Package`) and `parserFactory=...SqlBabelParserImpl#FACTORY`. Not tried. | +| `SELECT ... BY` (GROUP BY + ORDER BY shorthand) | 1.42 | Parse error | Same as above, presumably Babel. | +| Lambdas `x -> ...` | 1.37 | Parse error | Lambdas are accepted only as function arguments. The only higher-order function (Spark `EXISTS`) collides with the `EXISTS` keyword in the core parser. | +| Lateral column alias (`select f(x) r, r / 1024 kb`) | 1.40 | "Column 'r' not found" | Not enabled through the JDBC settings used here. | +| `MEASURE` columns | 1.38 | `AGGREGATE` not found | Not usable through this connection. | +| Table function in `FROM` without `TABLE()` | 1.37 | Parses, but `getMapEntries(m.this)` fails with "Table 'm' not found" | Correlated use still needs `lateral table(...)` or `cross apply table(...)`. | +| `LEFT_SHIFT(x, n)` function | 1.41 | Not found, with or without libraries | Use the `<<` operator. | + +--- + +## 4. Suggested follow-ups + +1. **Turn on a curated `fun` list** (start with `postgresql`) in `CalciteDataSource`. Mention upper-case names in + the SQL Reference tab, and add the functions to the schema browser's Functions group. +2. **Evaluate the Babel parser** for `SELECT * EXCLUDE/REPLACE`. Heap tables have many columns, so + `select * exclude ("@class", "@classLoader")` would be a real ergonomic win. Check that it does not break the + plugin's own syntax: `this['field']`, `instanceof.` names, `cross apply`. +3. **Add regression tests** for the ASOF join, `EXCLUDE CURRENT ROW` and bitwise queries above, so future Calcite + upgrades keep them working. diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a489d6..45f0a69 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,33 @@ ## [Unreleased] ### Added - SQL function `getHexAddress(x)`: the object address as `0x...`, the form MAT command lines and labels use. +- SQL date and time functions: `toDateTime(x[, zone])` shows heap date objects as e.g. `2026-10-01 11:11:11.224` + (java.util.Date, java.sql.Date/Time/Timestamp, Calendar, java.time Instant, LocalDate, LocalTime, LocalDateTime, + ZonedDateTime, OffsetDateTime, and numbers as epoch milliseconds). Also `toEpochMillis(x[, zone])`, + `millisToDateTime(long[, zone])`, `secondsToDateTime(long[, zone])` and `toDuration(x)`. Points in time default to + UTC, zoned values use their own zone, java.sql.Date/Time use the dumped JVM's zone; `'jvm'` selects that zone. +- `instanceof."java.lang.ref.Reference"`: the instanceof schema takes a class's full name as one quoted + identifier, like the root schema does, so package parts that are SQL words (`ref`, `user`) need no quoting. +- SQL editor: right-click menu with Run, Explain, Undo/Redo, Cut/Copy/Paste, Delete, Select All and + Toggle Comment. +- SQL editor: "SQL Reference" toolbar button that opens tables, virtual columns and functions in a separate + tab, so the current result stays open. The editor now starts with a short hint and a sample query instead of the 30-line help comment. +- SQL editor syntax colors are theme colors (Preferences > Appearance > Colors and Fonts > SQL (Calcite) + editor), with defaults for the dark theme. +- SQL editor keys (F5 / Ctrl+Enter, F10, Ctrl+Space, Ctrl+/) are now workbench commands in their own key + binding context. MAT binds F5, Ctrl+Enter and Ctrl+Space to its OQL commands, and the workbench consumed + those keys before the SQL editor could see them. +- SQL editor: schema browser beside the editor (toolbar toggle) listing classes by package, classes with + subclasses (instanceof), thread stacks and functions, with each class's columns and its this['@shallow'] / + this['@retained'] sizes. Double-click inserts the name at the cursor, quoted where SQL needs it; the filter box + lists matching tables. +- "SQL > Open in SQL Editor" on the context menu of objects and classes in MAT's views (histogram, dominator + tree, object lists...). It adds a query for the selection to the last used SQL tab (a class row selects all + instances, objects are selected by address, one statement per class) and selects it without running it. +- SQL editor: completion proposals pop up after typing '.' or '"', not only on Ctrl+Space. +- SQL result rows show MAT's object icons (instance, class, array, class loader, GC root overlay). +- Navigation history: re-opening a SQL result restores its query in the editor and runs it again, as the OQL + pane does. ### Changed - **BREAKING**: Eclipse Memory Analyzer 1.17.0 or higher is now required (previously 1.14.0). @@ -15,6 +42,8 @@ (`https://download.eclipse.org/mat/1.17.0/update-site/`) instead of an Eclipse simultaneous release, so the MAT API version matches the MAT users run. The URL names the release: `latest` would move to the next MAT release by itself. +- SQL errors are shown without adding an entry to the navigation history, as the OQL pane does. +- The SQL result area takes 4/5 of the pane height, as in the OQL pane. ### Fixed - `getDominator(x)` failed with "Object -1 not found" for objects dominated by the virtual root (e.g. shared @@ -42,6 +71,45 @@ - `testStackTraces` failed: MAT now renders a native frame with its source file, `(Object.java(Native Method))` instead of `(Native Method)`. That text comes from MAT unchanged, so the test ignores the source file part. +- SQL editor: an error in a query run from a selection (F5 on selected text, which "Open in SQL Editor" sets up) + or from Explain was underlined at the wrong place: the position counted from the start of the editor and + ignored the `explain plan for` prefix. A position outside the text threw instead, and the error was not shown. +- SQL editor: an error underline stayed after the query was fixed or another one ran; each run now clears it. +- SQL editor: Cancel in the progress view did not stop a running query, and closing the tab left it running. Both + now stop the heap scan. +- SQL editor: a query that finished after its tab was closed failed on the disposed editor. Closing the tab while + the editor had focus could leave its key bindings active and the Cut/Copy/Paste handlers pointing at it. +- "Open in SQL Editor" on a class row of a partial histogram (Show Retained Set, a dominator tree grouped by class) + selected every instance of the class, not just the objects of the row. +- "Open in SQL Editor" added the query to the active editor, which with several dumps open is not necessarily + the one the objects belong to; it now finds the editor of their dump. +- Schema browser filter: it sorted every table name of the dump on each keystroke (four per class, slow on dumps + with 100,000 classes). The sorted names are now kept per schema, and the search runs when typing pauses. +- `toDateTime(x)` returned the stale time of a `java.util.Date` changed with a deprecated setter (`setHours`...) + and of a `Calendar` changed with `set(...)` since its time was computed; it now returns null for them. +- `toDuration(x)` overflowed beyond about 292 years: `Duration.ofSeconds(Long.MAX_VALUE)`, a common "forever", + showed as `-0s`. +- Identifiers are quoted by Calcite's own reserved words, not only SQL-92's: the schema browser + wrote `java.lang.ref` unquoted, which does not parse (`REF` is reserved). +- The SQL reference used `instanceof.java.util.Map` as its example, which is always empty: heap + dumps record superclasses but not interfaces. It now uses `java.util.AbstractMap` and says why. +- Running a query could scroll the SQL editor when it was scrolled to the end: JFace's + ToolBarManager.relayout packs the toolbar's parent, which briefly lays the editor out taller, and StyledText + scrolls to fill the extra space. The editor now restores its scroll position after such a transient resize. +- Line numbers of the SQL editor use the editor's colors, so they are readable in the dark theme. +- Backspace, Delete and the arrow keys stopped working in the query editor after a table result was shown, while + letters still typed, until the window was deactivated and activated again. Cause: MAT shows table results with a + "Retrieving view elements..." user job whose progress dialog is opened and closed without ever getting focus. SWT + on GTK still records it as the active shell and, once it is disposed, has no active shell at all, so it drops every + key press that is not committed text. On Wayland the window gets no new focus-in event to repair that. The SQL + pane now re-activates the workbench window in that case (only if the window was never deactivated), which also + fixes MAT's own OQL pane. +- The SQL pane leaked its title image and syntax colors. The title image now comes from the shared JFace image + registry (the navigation history keeps showing it after the tab is closed), and the colors from the theme + registry. +- Undo and Redo keys (Ctrl+Z, Ctrl+Y) did not work in the SQL editor; the right-click menu's Cut and Copy could be + disabled while text was selected. +- Removed debug output printed to stdout on every result table refresh and explain plan. ### Notes - Only the MAT units this plugin needs are listed in the target (api, report, ui, parser, hprof). diff --git a/MatCalcitePlugin/plugin.xml b/MatCalcitePlugin/plugin.xml index 16b856c..abafe52 100644 --- a/MatCalcitePlugin/plugin.xml +++ b/MatCalcitePlugin/plugin.xml @@ -6,6 +6,9 @@ + + @@ -21,6 +24,146 @@ class="com.github.vlsi.mat.calcite.editor.CalcitePane" id="CALCITE"> + + + + + + + Syntax colors of the SQL (Calcite) query editor + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + false : listener::isCanceled)) { con = CalciteDataSource.getConnection(snapshot); st = con.createStatement(); @@ -61,7 +63,6 @@ public IResult execute(IProgressListener listener) throws Exception { && EXPLAIN_PLAN.matcher(sql).find()) { rowSet.absolute(1); String plan = rowSet.getString(1); - System.out.println("plan = " + plan); { Matcher ind = RENAME_INDEX.matcher(plan); StringBuffer sb = new StringBuffer(); diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/QueryCancellation.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/QueryCancellation.java new file mode 100644 index 0000000..6ef3d7b --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/QueryCancellation.java @@ -0,0 +1,51 @@ +package com.github.vlsi.mat.calcite; + +import java.util.concurrent.CancellationException; +import java.util.function.BooleanSupplier; + +/** + * Lets long scans stop when the query that runs them is cancelled. The caller marks the code it runs with + * {@link #enter}, and the heap object scan calls {@link #check} for each object. + * + *

This deliberately does not use {@link Thread#interrupt()}: MAT reads its index files through a + * {@link java.nio.channels.FileChannel}, which an interrupt closes for good, and an interrupt sent just as one call + * finishes would hit the next call on the same thread. A flag belongs to one call only. + */ +public final class QueryCancellation { + private static final ThreadLocal CURRENT = new ThreadLocal<>(); + + private QueryCancellation() { + } + + /** + * Runs the current thread's work under the given flag until the returned scope is closed. + * + * @param cancelled tells whether the work has been cancelled; called often, so it must be cheap + */ + public static Scope enter(BooleanSupplier cancelled) { + BooleanSupplier previous = CURRENT.get(); + CURRENT.set(cancelled); + return () -> { + if (previous == null) { + CURRENT.remove(); + } else { + CURRENT.set(previous); + } + }; + } + + /** Throws {@link CancellationException} if the current thread's work was cancelled. */ + public static void check() { + BooleanSupplier cancelled = CURRENT.get(); + if (cancelled != null && cancelled.getAsBoolean()) { + throw new CancellationException("Query cancelled"); + } + } + + /** The extent of {@link #enter}; closing it restores the previous flag. */ + @FunctionalInterface + public interface Scope extends AutoCloseable { + @Override + void close(); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/RowSetTable.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/RowSetTable.java index e21106c..1f1c00b 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/RowSetTable.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/RowSetTable.java @@ -3,16 +3,20 @@ import org.eclipse.mat.query.Column; import org.eclipse.mat.query.ContextProvider; import org.eclipse.mat.query.IContextObject; +import org.eclipse.mat.query.IIconProvider; import org.eclipse.mat.query.IResultTable; import org.eclipse.mat.query.ResultMetaData; +import org.eclipse.mat.snapshot.model.IObject; +import org.eclipse.mat.snapshot.query.Icons; +import java.net.URL; import java.sql.ResultSetMetaData; import java.sql.SQLException; import java.sql.Types; import javax.sql.rowset.CachedRowSet; -public class RowSetTable implements IResultTable { +public class RowSetTable implements IResultTable, IIconProvider { private final ResultMetaData metaData; private final CachedRowSet rowSet; @@ -91,6 +95,23 @@ public IContextObject getContext(final Object row) { return getContext(row, idColumnPosition); } + /** + * Icon of the object in the context column (instance, class, array, class loader, with a GC root overlay), + * the same icons MAT shows in its own object lists. + */ + @Override + public URL getIcon(Object row) { + if (idColumnPosition == -1 || !(row instanceof Object[])) { + return null; + } + Object[] data = (Object[]) row; + if (idColumnPosition >= data.length || !(data[idColumnPosition] instanceof HeapReference)) { + return null; + } + IObject o = ((HeapReference) data[idColumnPosition]).getIObject(); + return Icons.forObject(o.getSnapshot(), o.getObjectId()); + } + private static IContextObject getContext(final Object row, final int columnPosition) { if (!(row instanceof Object[])) { return null; @@ -127,7 +148,6 @@ public Object getRow(int rowId) { @Override public int getRowCount() { - System.out.println("size: " + rowSet.size()); return rowSet.size(); } } diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SchemaCatalog.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SchemaCatalog.java new file mode 100644 index 0000000..6f88a97 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SchemaCatalog.java @@ -0,0 +1,121 @@ +package com.github.vlsi.mat.calcite; + +import org.apache.calcite.schema.Schema; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.TreeSet; +import java.util.WeakHashMap; + +/** + * Finds the tables and functions of the heap schema ({@link CalciteDataSource#getHeapSchema}), for the schema browser. + */ +public final class SchemaCatalog { + /** Sub-schema of classes with their subclasses. */ + public static final String INSTANCEOF = "instanceof"; + /** Sub-schema of tables that are not classes, e.g. ThreadStackFrames. */ + public static final String NATIVE = "native"; + /** Prefix of internal tables that list object ids. */ + public static final String IDS_PREFIX = "$ids$:"; + + private SchemaCatalog() { + } + + /** A table, with the name to use in SQL. */ + public static final class TableRef { + /** Schema that holds the table. */ + public final Schema schema; + /** Name of the table inside {@link #schema}. */ + public final String tableName; + /** Readable full name, e.g. {@code java.util.HashMap} or {@code native.ThreadStackFrames}. */ + public final String fullName; + /** Name as written in SQL, quoted where needed. */ + public final String sqlName; + + TableRef(Schema schema, String tableName, String fullName, String sqlName) { + this.schema = schema; + this.tableName = tableName; + this.fullName = fullName; + this.sqlName = sqlName; + } + } + + /** + * Classes whose full name contains the filter (ignoring case), then matching tables that are not classes, sorted + * by name. An empty filter matches everything. Returns at most {@code limit} tables. + */ + public static List findTables(Schema root, String filter, int limit) { + String lower = filter == null ? "" : filter.trim().toLowerCase(Locale.ROOT); + List result = new ArrayList<>(); + ClassNames classes = classNames(root); + for (int i = 0; i < classes.names.length; i++) { + if (result.size() >= limit) { + return result; + } + if (classes.lowerCase[i].contains(lower)) { + // The root schema lists every class by its full name, usable as a single quoted identifier + String name = classes.names[i]; + result.add(new TableRef(root, name, name, SqlIdentifiers.quote(name))); + } + } + Schema nativeSchema = root.getSubSchema(NATIVE); + if (nativeSchema != null) { + for (String name : new TreeSet<>(nativeSchema.getTableNames())) { + if (result.size() >= limit) { + return result; + } + String fullName = NATIVE + "." + name; + if (fullName.toLowerCase(Locale.ROOT).contains(lower)) { + result.add(new TableRef(nativeSchema, name, fullName, SqlIdentifiers.quotePath(fullName))); + } + } + } + return result; + } + + /** + * The class tables of each schema, sorted, with their lower-case names. The schema browser searches them on every + * keystroke, and the root schema lists about four tables per class (plain, instanceof., $ids$: ...): sorting them + * each time took long for dumps with 100,000 classes. A schema's tables never change. + */ + private static final Map CLASS_NAMES = Collections.synchronizedMap(new WeakHashMap<>()); + + private static final class ClassNames { + final String[] names; + final String[] lowerCase; + + ClassNames(List sorted) { + names = sorted.toArray(new String[0]); + lowerCase = new String[names.length]; + for (int i = 0; i < names.length; i++) { + lowerCase[i] = names[i].toLowerCase(Locale.ROOT); + } + } + } + + private static ClassNames classNames(Schema root) { + return CLASS_NAMES.computeIfAbsent(root, schema -> { + List names = new ArrayList<>(); + for (String name : new TreeSet<>(schema.getTableNames())) { + if (!name.startsWith(IDS_PREFIX) && !name.startsWith(INSTANCEOF + ".")) { + names.add(name); + } + } + return new ClassNames(names); + }); + } + + /** Names of the SQL functions, sorted, without internal ones. */ + public static List functionNames(Schema root) { + List names = new ArrayList<>(); + for (String name : new TreeSet<>(root.getFunctionNames())) { + if (!"getHints".equals(name)) { // internal, used by content assist + names.add(name); + } + } + return names; + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlForObjects.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlForObjects.java new file mode 100644 index 0000000..f8e3954 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlForObjects.java @@ -0,0 +1,91 @@ +package com.github.vlsi.mat.calcite; + +import org.eclipse.mat.SnapshotException; +import org.eclipse.mat.query.IContextObject; +import org.eclipse.mat.query.IContextObjectSet; +import org.eclipse.mat.snapshot.ISnapshot; +import org.eclipse.mat.snapshot.model.IClass; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Builds SQL that selects the objects chosen in another MAT view: a whole class for a class row (e.g. in the + * histogram), otherwise the objects themselves, by address. Objects of several classes give one statement per + * class, since their columns differ. + */ +public final class SqlForObjects { + /** Keeps the statement small enough to edit and fast enough to plan. */ + public static final int MAX_OBJECTS_PER_CLASS = 1000; + + private SqlForObjects() { + } + + public static String build(ISnapshot snapshot, List elements) throws SnapshotException { + Set wholeClasses = new LinkedHashSet<>(); + Map> addressesByClass = new LinkedHashMap<>(); + Map countByClass = new LinkedHashMap<>(); + + for (IContextObject element : elements) { + int id = element.getObjectId(); + int[] ids = element instanceof IContextObjectSet ? ((IContextObjectSet) element).getObjectIds() : new int[]{id}; + if (element instanceof IContextObjectSet && id >= 0 && snapshot.isClass(id)) { + IClass cls = (IClass) snapshot.getObject(id); + // A class row of the full histogram stands for all its instances. A row of a partial histogram (retained + // set, grouped dominator tree...) stands for some of them only, so those are selected by address. + if (ids.length == cls.getNumberOfObjects()) { + wholeClasses.add(cls.getName()); + continue; + } + } + for (int objectId : ids) { + if (objectId < 0) { + continue; + } + String className = snapshot.getClassOf(objectId).getName(); + int count = countByClass.merge(className, 1, Integer::sum); + if (count <= MAX_OBJECTS_PER_CLASS) { + addressesByClass.computeIfAbsent(className, k -> new ArrayList<>()) + .add(snapshot.mapIdToAddress(objectId)); + } + } + } + + List statements = new ArrayList<>(); + for (String className : wholeClasses) { + statements.add("select *\n from " + SqlIdentifiers.quote(className) + "\n"); + } + for (Map.Entry> entry : addressesByClass.entrySet()) { + String className = entry.getKey(); + StringBuilder sb = new StringBuilder(); + int total = countByClass.get(className); + if (total > MAX_OBJECTS_PER_CLASS) { + sb.append("-- First ").append(MAX_OBJECTS_PER_CLASS).append(" of ").append(total).append(" objects\n"); + } + sb.append("select *\n from ").append(SqlIdentifiers.quote(className)) + .append("\n where getAddress(this) in ("); + List addresses = entry.getValue(); + for (int i = 0; i < addresses.size(); i++) { + if (i > 0) { + sb.append(i % 8 == 0 ? ",\n " : ", "); + } + sb.append(addresses.get(i)); + } + sb.append(")\n"); + statements.add(sb.toString()); + } + + if (statements.isEmpty()) { + throw new SnapshotException("No heap objects selected"); + } + if (statements.size() == 1) { + return statements.get(0); + } + return "-- Objects of several classes: select one statement and press F5 to run it\n\n" + + String.join("\n", statements); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlIdentifiers.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlIdentifiers.java new file mode 100644 index 0000000..f32dd7a --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlIdentifiers.java @@ -0,0 +1,46 @@ +package com.github.vlsi.mat.calcite; + +import org.apache.calcite.sql.parser.SqlAbstractParserImpl; +import org.apache.calcite.sql.parser.SqlParser; + +import java.util.Locale; +import java.util.regex.Pattern; + +/** + * Writes identifiers for the SQL dialect of {@link CalciteDataSource#getConnection}: Calcite's JAVA lexer, which is + * case-sensitive and quotes with double quotes. + */ +public final class SqlIdentifiers { + private static final Pattern PLAIN_IDENTIFIER = Pattern.compile("[A-Za-z_][A-Za-z0-9_]*"); + /** Calcite's parser reserves more than SQL-92 does, e.g. REF, so java.lang.ref must be written java.lang."ref". */ + private static final SqlAbstractParserImpl.Metadata PARSER = SqlParser.create("").getMetadata(); + + private SqlIdentifiers() { + } + + /** + * Quotes an identifier unless it is a plain identifier that is not a reserved word, e.g. {@code size} stays as is, + * {@code @classLoader} becomes {@code "@classLoader"} and {@code java.lang.String} becomes + * {@code "java.lang.String"} (the heap schema lists every class by its full name too). + */ + public static String quote(String identifier) { + if (PLAIN_IDENTIFIER.matcher(identifier).matches() && !PARSER.isReservedWord(identifier.toUpperCase(Locale.ROOT))) { + return identifier; + } + return '"' + identifier.replace("\"", "\"\"") + '"'; + } + + /** + * Quotes each part of a dotted schema path where needed, e.g. {@code java.util."HashMap$Node"}. + */ + public static String quotePath(String path) { + StringBuilder sb = new StringBuilder(); + for (String part : path.split("\\.")) { + if (sb.length() > 0) { + sb.append('.'); + } + sb.append(quote(part)); + } + return sb.toString(); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlReference.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlReference.java new file mode 100644 index 0000000..94c00dd --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/SqlReference.java @@ -0,0 +1,61 @@ +package com.github.vlsi.mat.calcite; + +/** + * Quick reference of the SQL dialect: shown by the SQL editor's "SQL Reference" tab. + */ +public final class SqlReference { + public static final String TEXT = "SQL (Calcite) quick reference\n" + + "\n" + + "Keys:\n" + + " F5, Ctrl+Enter run the selected text, or the whole editor if nothing is selected\n" + + " F10 explain plan (same as prefixing the query with: explain plan for)\n" + + " Ctrl+Space complete table, column and function names\n" + + " Ctrl+/ comment or uncomment the current line\n" + + "\n" + + "Tables:\n" + + " \"java.math.BigInteger\" all BigIntegers\n" + + " instanceof.\"java.util.AbstractMap\" AbstractMap and every class that extends it\n" + + " (superclasses only: heap dumps do not record interfaces,\n" + + " so instanceof.\"java.util.Map\" is empty)\n" + + " java.util.\"HashMap$Node\" the same as \"java.util.HashMap$Node\", one schema per package\n" + + " instanceof.java.lang.\"ref\".Reference package form: quote parts that are SQL words (ref, user...)\n" + + " \"java.util.Date\" quote the whole name when a part is an SQL word: Date\n" + + "\n" + + "Virtual columns:\n" + + " this['@shallow'] shallow heap size\n" + + " this['@retained'] retained heap size\n" + + " this['@class'] class\n" + + " this['@className'] class name\n" + + " this['@super'] superclass\n" + + " this['@classLoader'] class loader\n" + + " this['fieldName'] the same as getField(this, 'fieldName')\n" + + " this['a.b.c'] the same as this['a']['b']['c']\n" + + "\n" + + "Functions:\n" + + " toString(any) string representation\n" + + " getAddress(any) address of the referenced object\n" + + " getHexAddress(any) the same as hex text, e.g. 0x7bfeb5848\n" + + " getDominator(any) immediate dominator, or null when dominated by the root\n" + + " getType(any) class name of the referenced object\n" + + " shallowSize(any) shallow heap size\n" + + " retainedSize(any) retained heap size\n" + + " length(array) length of an array reference\n" + + " getSize(collection|map|array)\n" + + " size of a collection or map, or number of non-null array elements\n" + + " getByKey(map, key) map element for the key with the given string representation\n" + + " getField(any, field name) value of the field for the given object\n" + + "\n" + + "Dates and times (java.util.Date, java.sql.Date/Time/Timestamp, Calendar, java.time.Instant, LocalDate,\n" + + "LocalTime, LocalDateTime, ZonedDateTime, OffsetDateTime, Duration, and numbers as epoch milliseconds):\n" + + " toDateTime(any) readable text, e.g. 2026-10-01 11:11:11.224; points in time in UTC,\n" + + " Calendar/Zoned/OffsetDateTime in their own zone (named after the time)\n" + + " toDateTime(any, zone) points in time in a zone, e.g. toDateTime(this, 'Europe/Madrid');\n" + + " 'jvm' is the dumped JVM's default zone (java.sql.Date/Time use it)\n" + + " toEpochMillis(any[, zone]) epoch milliseconds; local date-times taken in the zone (default UTC)\n" + + " millisToDateTime(long[, zone]), secondsToDateTime(long[, zone]) epoch number to readable text\n" + + " toDuration(any) java.time.Duration or milliseconds as e.g. 1d 2h 3m 4.567s\n" + + " Note: Date is an SQL keyword, so quote the table: select toDateTime(this) from \"java.util.Date\"\n"; + + private SqlReference() { + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/CalciteJob.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/CalciteJob.java index b8314b8..5aa051a 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/CalciteJob.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/CalciteJob.java @@ -9,10 +9,11 @@ import org.eclipse.mat.query.registry.QueryDescriptor; import org.eclipse.mat.query.registry.QueryRegistry; import org.eclipse.mat.query.registry.QueryResult; -import org.eclipse.mat.query.results.TextResult; import org.eclipse.mat.ui.editor.AbstractPaneJob; import org.eclipse.mat.ui.util.PaneState; import org.eclipse.mat.ui.util.ProgressMonitorWrapper; +import org.eclipse.swt.widgets.Display; +import org.eclipse.ui.PlatformUI; import java.io.PrintWriter; import java.io.StringWriter; @@ -22,11 +23,20 @@ public class CalciteJob extends AbstractPaneJob { private final String sql; private final CalcitePane calcitePane; private final PaneState state; + /** Where sql starts in the editor, after a prefix of this length that is not in the editor, e.g. explain plan for. */ + private final int editorOffset; + private final int prefixLength; public CalciteJob(String sql, CalcitePane pane, PaneState state) { + this(sql, pane, state, 0, 0); + } + + public CalciteJob(String sql, CalcitePane pane, PaneState state, int editorOffset, int prefixLength) { super(sql, pane); this.sql = sql; this.state = state; + this.editorOffset = editorOffset; + this.prefixLength = prefixLength; calcitePane = pane; this.setUser(true); } @@ -41,10 +51,21 @@ protected IStatus doRun(IProgressMonitor monitor) { argumentSet.setArgumentValue("sql", sql);//$NON-NLS-1$ final QueryResult result = argumentSet .execute(new ProgressMonitorWrapper(monitor)); - calcitePane.getQueryString().getDisplay().asyncExec(() -> calcitePane.initQueryResult(result, state)); + // The tab may have been closed while the query ran: then there is nothing to show the result in + display().asyncExec(() -> { + if (!calcitePane.getQueryString().isDisposed()) { + calcitePane.initQueryResult(result, state, sql); + } + }); } catch (final Throwable e) { - calcitePane.getQueryString().getDisplay().asyncExec(() -> { - String keyError = calcitePane.highlightError(e); + if (monitor.isCanceled()) { + return Status.CANCEL_STATUS; // cancelled from the progress view, or the tab was closed + } + display().asyncExec(() -> { + if (calcitePane.getQueryString().isDisposed()) { + return; + } + String keyError = calcitePane.highlightError(e, sql, editorOffset, prefixLength); StringWriter sw = new StringWriter(); if (keyError != null) { @@ -54,11 +75,8 @@ protected IStatus doRun(IProgressMonitor monitor) { sw.append(sql).append('\n'); } e.printStackTrace(new PrintWriter(sw)); - String exceptionText = sw.toString(); - - TextResult tr = new TextResult(exceptionText, false); - QueryResult result = new QueryResult(descriptor, "calcite", tr); - calcitePane.initQueryResult(result, state); + // Like the OQL pane, errors are shown without a navigation history entry + calcitePane.showText(sw.toString()); }); e.printStackTrace(); } @@ -66,4 +84,14 @@ protected IStatus doRun(IProgressMonitor monitor) { return Status.OK_STATUS; } + /** The jobs of a pane, which it cancels when it is closed. */ + @Override + public boolean belongsTo(Object family) { + return family == calcitePane || super.belongsTo(family); + } + + private static Display display() { + return PlatformUI.getWorkbench().getDisplay(); + } + } diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/ExecuteQueryAction.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/ExecuteQueryAction.java index f00ea5d..d867510 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/ExecuteQueryAction.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/action/ExecuteQueryAction.java @@ -27,28 +27,26 @@ public ExecuteQueryAction(CalcitePane pane, PaneState state, boolean doExplain) "/explain.png" : "resources/icons/run.png")); } - private String getSelectedQuery() { + @Override + public void run() { StyledText queryString = pane.getQueryString(); String query = queryString.getSelectionText(); - + // Where the SQL that runs starts in the editor, so an error position can be shown at the right place + int offset = queryString.getSelection().x; if ("".equals(query)) //$NON-NLS-1$ { query = queryString.getText(); + offset = 0; } - // Temporary workaround for https://issues.apache.org/jira/browse/CALCITE-459 query = query + queryString.getLineDelimiter(); - return query; - } - - @Override - public void run() { - String query = getSelectedQuery(); //TODO: actually check with this regexp doesn't work if there are comments in the beginning of the query + String prefix = ""; if (doExplain && !STARTS_WITH_EXPLAIN_PLAN.matcher(query).find()) { - query = "explain plan for " + query; + prefix = "explain plan for "; } - new CalciteJob(query, pane, state).schedule(); + pane.clearErrorHighlight(); + new CalciteJob(prefix + query, pane, state, offset, prefix.length()).schedule(); } } diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/ActiveShellFix.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/ActiveShellFix.java new file mode 100644 index 0000000..2afca85 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/ActiveShellFix.java @@ -0,0 +1,75 @@ +package com.github.vlsi.mat.calcite.editor; + +import org.eclipse.swt.SWT; +import org.eclipse.swt.widgets.Display; +import org.eclipse.swt.widgets.Listener; +import org.eclipse.swt.widgets.Shell; + +/** + * Works around SWT on GTK losing track of the active window, which makes Backspace, Delete and the arrow keys stop + * working in the query editor while letters still type. + * + *

When MAT shows a table result it runs a "Retrieving view elements..." user job. Eclipse opens a progress dialog + * for it, and when the job finishes quickly the dialog is opened and closed within the same call, without ever + * getting keyboard focus. Opening it still makes SWT record it as the active shell, and closing it sets + * {@code Display.getActiveShell()} to null. Only a new focus-in event on the workbench window restores it. The window + * never lost focus, so on Wayland that event does not come. While the active shell is null, + * {@code Display.getFocusControl()} is null and SWT drops every key press that is not committed text. + * + *

The fix tracks whether the workbench window itself was deactivated. If a child shell is disposed and leaves no + * active shell while the window was never deactivated, the window still has focus, so it is re-activated. If the + * user switched to another application in between, the window got a Deactivate event and nothing is done. + */ +final class ActiveShellFix { + private static final String KEY = ActiveShellFix.class.getName(); + + private final Shell shell; + private boolean shellActive; + + private ActiveShellFix(Shell shell) { + this.shell = shell; + this.shellActive = shell.getDisplay().getActiveShell() == shell; + } + + /** + * Installs the fix for the given workbench window, once per window. + */ + static void install(Shell shell) { + if (shell.getData(KEY) != null) { + return; + } + ActiveShellFix fix = new ActiveShellFix(shell); + shell.setData(KEY, fix); + Display display = shell.getDisplay(); + Listener listener = fix::handleEvent; + display.addFilter(SWT.Activate, listener); + display.addFilter(SWT.Deactivate, listener); + display.addFilter(SWT.Dispose, listener); + shell.addDisposeListener(e -> { + display.removeFilter(SWT.Activate, listener); + display.removeFilter(SWT.Deactivate, listener); + display.removeFilter(SWT.Dispose, listener); + }); + } + + private void handleEvent(org.eclipse.swt.widgets.Event e) { + if (e.widget == shell) { + if (e.type == SWT.Activate) { + shellActive = true; + } else if (e.type == SWT.Deactivate) { + shellActive = false; + } + return; + } + if (e.type != SWT.Dispose || !(e.widget instanceof Shell) || ((Shell) e.widget).getParent() != shell) { + return; + } + // The active shell is cleared while the child shell is released, after this event, so check afterwards + Display display = shell.getDisplay(); + display.asyncExec(() -> { + if (!shell.isDisposed() && shellActive && display.getActiveShell() == null) { + shell.forceActive(); + } + }); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteContentAssistantProcessor.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteContentAssistantProcessor.java index 46916d3..0162adb 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteContentAssistantProcessor.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteContentAssistantProcessor.java @@ -148,7 +148,8 @@ public char[] getCompletionProposalAutoActivationCharacters() { @Override public char[] getContextInformationAutoActivationCharacters() { - return new char[]{' ', '(', '.'}; + // No context information is provided (computeContextInformation returns nothing) + return null; } @Override diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalcitePane.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalcitePane.java index 34b412d..d7f5306 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalcitePane.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalcitePane.java @@ -7,15 +7,21 @@ import org.apache.calcite.sql.parser.SqlParseException; import org.apache.calcite.sql.parser.SqlParserPos; import org.apache.calcite.sql.parser.SqlParserUtil; -import org.eclipse.core.commands.operations.IUndoContext; +import org.eclipse.core.runtime.jobs.Job; import org.eclipse.jface.action.Action; +import org.eclipse.jface.action.IAction; +import org.eclipse.jface.action.IMenuManager; import org.eclipse.jface.action.IToolBarManager; +import org.eclipse.jface.action.MenuManager; +import org.eclipse.jface.action.Separator; import org.eclipse.jface.preference.JFacePreferences; +import org.eclipse.jface.resource.ImageRegistry; import org.eclipse.jface.resource.JFaceResources; +import org.eclipse.jface.text.BadLocationException; import org.eclipse.jface.text.Document; import org.eclipse.jface.text.IDocument; import org.eclipse.jface.text.IDocumentPartitioner; -import org.eclipse.jface.text.IUndoManagerExtension; +import org.eclipse.jface.text.ITextOperationTarget; import org.eclipse.jface.text.rules.FastPartitioner; import org.eclipse.jface.text.source.CompositeRuler; import org.eclipse.jface.text.source.ISourceViewer; @@ -24,9 +30,12 @@ import org.eclipse.jface.text.source.SourceViewerConfiguration; import org.eclipse.mat.query.IResult; import org.eclipse.mat.query.registry.QueryResult; +import org.eclipse.mat.snapshot.ISnapshot; import org.eclipse.mat.ui.editor.AbstractEditorPane; import org.eclipse.mat.ui.editor.CompositeHeapEditorPane; import org.eclipse.mat.ui.editor.EditorPaneRegistry; +import org.eclipse.mat.ui.editor.MultiPaneEditor; +import org.eclipse.mat.ui.util.ErrorHelper; import org.eclipse.mat.ui.util.PaneState; import org.eclipse.mat.ui.util.PaneState.PaneType; import org.eclipse.swt.SWT; @@ -37,46 +46,84 @@ import org.eclipse.swt.events.FocusListener; import org.eclipse.swt.events.KeyAdapter; import org.eclipse.swt.events.KeyEvent; +import org.eclipse.swt.graphics.Color; import org.eclipse.swt.graphics.Image; +import org.eclipse.swt.graphics.Point; +import org.eclipse.swt.graphics.RGB; import org.eclipse.swt.widgets.Composite; +import org.eclipse.swt.widgets.Display; import org.eclipse.ui.IActionBars; -import org.eclipse.ui.IEditorInput; -import org.eclipse.ui.IEditorSite; -import org.eclipse.ui.IWorkbenchCommandConstants; +import org.eclipse.ui.ISharedImages; import org.eclipse.ui.IWorkbenchWindow; -import org.eclipse.ui.PartInitException; +import org.eclipse.ui.PlatformUI; import org.eclipse.ui.actions.ActionFactory; -import org.eclipse.ui.operations.RedoActionHandler; -import org.eclipse.ui.operations.UndoActionHandler; +import org.eclipse.ui.contexts.IContextActivation; +import org.eclipse.ui.contexts.IContextService; import org.eclipse.ui.plugin.AbstractUIPlugin; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + public class CalcitePane extends CompositeHeapEditorPane { + /** Key binding context active while the SQL editor has focus, see plugin.xml. */ + public static final String CONTEXT_ID = "com.github.vlsi.mat.calcite.sqlEditor"; + private static final String COMMAND_PREFIX = "com.github.vlsi.mat.calcite."; + private static final String TITLE_IMAGE_KEY = "com.github.vlsi.mat.calcite.paneImage"; + + private static final String HINT = "-- F5 or Ctrl+Enter: run, F10: explain plan, Ctrl+Space: complete," + + " Ctrl+/: comment line\n" + + "-- Tables, columns and functions: see the SQL Reference button in the toolbar\n"; + private static final String DEFAULT_QUERY = HINT + + "select u.this, retainedSize(s.this) retained_size\n" + + " from \"java.lang.String\" s\n" + + " , \"java.net.URL\" u\n" + + " where s.this = u.path\n"; + + /** Open SQL panes, least recently used first, see {@link #lastUsed(MultiPaneEditor)}. UI thread only. */ + private static final List OPEN_PANES = new ArrayList<>(); + private SourceViewer queryViewer; private StyledText queryString; private ExecuteQueryAction executeQueryAction; private ExecuteQueryAction explainQueryAction; - private Action copyQueryStringAction; private Action commentLineAction; private Action contentAssistAction; + private Action showReferenceAction; + private Action toggleSchemaAction; + private SashForm outerSash; + private SchemaBrowser schemaBrowser; + /** Text editing actions for the context menu and the global Cut/Copy/Paste handlers, keyed by action factory id. */ + private final Map textActions = new LinkedHashMap<>(); + private TextOperationAction undoAction; + private TextOperationAction redoAction; + private IContextActivation contextActivation; public CalcitePane() { } @Override public void createPartControl(Composite parent) { - SashForm sash = new SashForm(parent, SWT.VERTICAL | SWT.SMOOTH); + OPEN_PANES.add(this); + // Schema browser on the left, full height; query editor above the results on the right + outerSash = new SashForm(parent, SWT.HORIZONTAL | SWT.SMOOTH); + schemaBrowser = new SchemaBrowser(outerSash, getSnapshot(), this::insertText); + SashForm sash = new SashForm(outerSash, SWT.VERTICAL | SWT.SMOOTH); int VERTICAL_RULER_WIDTH = 12; CompositeRuler ruler = new CompositeRuler(VERTICAL_RULER_WIDTH); - ruler.addDecorator(0, new LineNumberRulerColumn()); + ruler.addDecorator(0, new ThemedLineNumberRulerColumn()); queryViewer = new SourceViewer(sash, ruler, SWT.H_SCROLL | SWT.V_SCROLL | SWT.MULTI); - queryViewer.configure(new CalciteSourceViewerConfiguration()); queryString = queryViewer.getTextWidget(); // The following setBackground(getBackground) results in proper white background in MACOS. // No sure why the background is gray otherwise. queryString.setBackground(queryString.getBackground()); queryString.setFont(JFaceResources.getFont(JFaceResources.TEXT_FONT)); + TransientResizeScrollFix.install(queryString); + // Fallback for hosts where the key bindings from plugin.xml are not active (they are declared in MAT's key + // binding scheme). When a binding handles a key, the workbench consumes the event and these listeners never see it. queryString.addTraverseListener(e -> { if (e.detail == SWT.TRAVERSE_RETURN && (e.stateMask & SWT.MOD1) != 0) { executeQueryAction.run(); @@ -102,18 +149,40 @@ public void keyPressed(KeyEvent e) { } }); - this.queryString.addFocusListener(new FocusListener() { + ActiveShellFix.install(queryString.getShell()); + queryString.addListener(SWT.MouseDown, e -> reactivateShell()); + queryString.addFocusListener(new FocusListener() { + @Override public void focusGained(FocusEvent e) { - IActionBars actionBars = CalcitePane.this.getEditor().getEditorSite().getActionBars(); - actionBars.setGlobalActionHandler(ActionFactory.COPY.getId(), CalcitePane.this.copyQueryStringAction); + OPEN_PANES.remove(CalcitePane.this); + OPEN_PANES.add(CalcitePane.this); + reactivateShell(); + IContextService contextService = getEditor().getSite().getService(IContextService.class); + if (contextService != null && contextActivation == null) { + contextActivation = contextService.activateContext(CONTEXT_ID); + } + IActionBars actionBars = getEditor().getEditorSite().getActionBars(); + for (Map.Entry entry : textActions.entrySet()) { + actionBars.setGlobalActionHandler(entry.getKey(), entry.getValue()); + } actionBars.updateActionBars(); } + @Override public void focusLost(FocusEvent e) { + if (contextActivation != null) { + contextActivation.getContextService().deactivateContext(contextActivation); + contextActivation = null; + } + IActionBars actionBars = getEditor().getEditorSite().getActionBars(); + for (String id : textActions.keySet()) { + actionBars.setGlobalActionHandler(id, null); + } + actionBars.updateActionBars(); } }); - IDocument doc = createDocument(); + IDocument doc = new Document(DEFAULT_QUERY); SourceViewerConfiguration svc = new CalciteSourceViewerConfiguration(); IDocumentPartitioner partitioner = new FastPartitioner( new CalcitePartitionScanner(), @@ -126,69 +195,29 @@ public void focusLost(FocusEvent e) { queryString.selectAll(); createContainer(sash); - makeActions(); - - installUndoRedoSupport(); - } - - private void installUndoRedoSupport() { - IUndoContext undoContext = ((IUndoManagerExtension) queryViewer.getUndoManager()).getUndoContext(); + sash.setWeights(1, 4); + outerSash.setWeights(1, 4); - UndoActionHandler undoAction = new UndoActionHandler(getSite(), undoContext); - RedoActionHandler redoAction = new RedoActionHandler(getSite(), undoContext); - - undoAction.setActionDefinitionId(IWorkbenchCommandConstants.EDIT_UNDO); - redoAction.setActionDefinitionId(IWorkbenchCommandConstants.EDIT_REDO); - - IActionBars actionBars = getEditor().getEditorSite().getActionBars(); - actionBars.setGlobalActionHandler(ActionFactory.UNDO.getId(), undoAction); - actionBars.setGlobalActionHandler(ActionFactory.REDO.getId(), redoAction); - - actionBars.updateActionBars(); - } - - private IDocument createDocument() { - IDocument doc = new Document(); - doc.set("-- explain plan for -- or F10\n" - + "-- Tables:\n" - + "-- java.math.BigInteger list of all BigIntegers\n" - + "-- instanceof.java.math.BigInteger BigIntegers and all subclasses\n" - + "-- instanceof.java.lang.\"HashMap$Entry\" Entry and all subclasses\n" - + "-- Virtual columns:\n" - + "-- this['@shallow'] -- returns shallow heap size\n" - + "-- this['@retained'] -- returns retained heap size\n" - + "-- this['@class'] -- returns class\n" - + "-- this['@className'] -- returns class name\n" - + "-- this['@super'] -- returns superclass\n" - + "-- this['@classLoader'] -- returns classLoader\n" - + "-- this['fieldName'] -- the same as getField(this, 'fieldName')\n" - + "-- this['a.b.c'] -- the same as this['a']['b']['c']\n" - + "-- Functions:\n" - + "-- toString(any) returns string representation\n" - + "-- getAddress(any) returns address of referenced object\n" - + "-- getType(any) returns class name of referenced object\n" - + "-- shallowSize(any) returns shallow heap size\n" - + "-- retainedSize(any) returns retained heap size\n" - + "-- length(array) returns length of array reference\n" - + "-- getSize(collection or map or array) returns size of collection or array, or number of non-null " + - "elements in array\n" - + "-- getByKey(map, key) returns element of map for key with given string representation\n" - + "-- getField(any, field name) returns value of field for given object\n" - + "select u.this, retainedSize(s.this) retained_size\n" - + " from \"java.lang.String\" s\n" - + " , \"java.net.URL\" u\n" - + " where s.this = u.path\n"); - return doc; + makeActions(); + hookContextMenu(); + queryViewer.addSelectionChangedListener(e -> updateTextActions()); + queryViewer.addTextListener(e -> updateTextActions()); + updateTextActions(); } - @Override - public void init(IEditorSite site, IEditorInput input) - throws PartInitException { - super.init(site, input); + /** + * Fallback for {@link ActiveShellFix}: when the user clicks or focuses the editor, the window is certainly active, + * so restore SWT's active shell if it was lost. A no-op while SWT's state is correct. + */ + private void reactivateShell() { + if (queryString.getDisplay().getActiveShell() == null) { + queryString.getShell().forceActive(); + } } @Override public void setFocus() { + queryString.setFocus(); } @Override @@ -196,36 +225,300 @@ public String getTitle() { return "Calcite"; } + /** + * The image is shared and never disposed: the navigation history keeps showing it for queries of SQL tabs that + * have already been closed. + */ @Override public Image getTitleImage() { - return AbstractUIPlugin.imageDescriptorFromPlugin("MatCalcitePlugin", "resources/icons/plugin.png").createImage(); + ImageRegistry registry = JFaceResources.getImageRegistry(); + Image image = registry.get(TITLE_IMAGE_KEY); + if (image == null) { + registry.put(TITLE_IMAGE_KEY, + AbstractUIPlugin.imageDescriptorFromPlugin("MatCalcitePlugin", "resources/icons/plugin.png")); + image = registry.get(TITLE_IMAGE_KEY); + } + return image; + } + + /** + * Runs an SQL editor command declared in plugin.xml, see {@link SqlEditorCommandHandler}. + */ + void runCommand(String commandId) { + switch (commandId.substring(COMMAND_PREFIX.length())) { + case "execute": + executeQueryAction.run(); + break; + case "explain": + explainQueryAction.run(); + break; + case "contentAssist": + contentAssistAction.run(); + break; + case "toggleComment": + commentLineAction.run(); + break; + default: + break; + } } private void makeActions() { executeQueryAction = new ExecuteQueryAction(this, null, false); explainQueryAction = new ExecuteQueryAction(this, null, true); commentLineAction = new CommentLineAction(queryString); - IWorkbenchWindow window = this.getEditorSite().getWorkbenchWindow(); - ActionFactory.IWorkbenchAction globalAction = ActionFactory.COPY.create(window); - this.copyQueryStringAction = new Action() { + commentLineAction.setText("Toggle Comment"); + contentAssistAction = new Action() { + @Override public void run() { - CalcitePane.this.queryString.copy(); + queryViewer.doOperation(ISourceViewer.CONTENTASSIST_PROPOSALS); } }; - this.copyQueryStringAction.setAccelerator(globalAction.getAccelerator()); - this.contentAssistAction = new Action() { + toggleSchemaAction = new Action("Schema Browser", IAction.AS_CHECK_BOX) { @Override public void run() { - queryViewer.doOperation(ISourceViewer.CONTENTASSIST_PROPOSALS); + outerSash.setMaximizedControl(isChecked() ? null : outerSash.getChildren()[1]); + } + }; + toggleSchemaAction.setToolTipText("Show or hide the tables, columns and functions"); + toggleSchemaAction.setImageDescriptor(PlatformUI.getWorkbench().getSharedImages() + .getImageDescriptor(ISharedImages.IMG_OBJ_FOLDER)); + toggleSchemaAction.setChecked(true); + showReferenceAction = new Action("SQL Reference") { + @Override + public void run() { + // A separate tab, so the current result stays where it is + getEditor().addNewPage(SqlReferencePane.ID, null, true); } }; + showReferenceAction.setToolTipText("SQL Reference: tables, virtual columns and functions"); + showReferenceAction.setImageDescriptor(PlatformUI.getWorkbench().getSharedImages() + .getImageDescriptor(ISharedImages.IMG_LCL_LINKTO_HELP)); + + undoAction = addTextAction(ActionFactory.UNDO, ITextOperationTarget.UNDO, true); + redoAction = addTextAction(ActionFactory.REDO, ITextOperationTarget.REDO, true); + addTextAction(ActionFactory.CUT, ITextOperationTarget.CUT, true); + addTextAction(ActionFactory.COPY, ITextOperationTarget.COPY, true); + addTextAction(ActionFactory.PASTE, ITextOperationTarget.PASTE, true); + addTextAction(ActionFactory.DELETE, ITextOperationTarget.DELETE, true); + addTextAction(ActionFactory.SELECT_ALL, ITextOperationTarget.SELECT_ALL, true); + } + + /** + * Creates a context menu action that runs a text operation on the query viewer, copying text, icon and + * accelerator from the matching workbench action. + * + * @param global whether the action also handles the workbench action (and its key binding) while the editor + * has focus + */ + private TextOperationAction addTextAction(ActionFactory factory, int operation, boolean global) { + IWorkbenchWindow window = getEditorSite().getWorkbenchWindow(); + ActionFactory.IWorkbenchAction globalAction = factory.create(window); + TextOperationAction action = new TextOperationAction(operation); + action.setText(globalAction.getText()); + action.setToolTipText(globalAction.getToolTipText()); + action.setImageDescriptor(globalAction.getImageDescriptor()); + action.setDisabledImageDescriptor(globalAction.getDisabledImageDescriptor()); + action.setActionDefinitionId(globalAction.getActionDefinitionId()); + globalAction.dispose(); + if (global) { + textActions.put(factory.getId(), action); + } + return action; + } + + private void hookContextMenu() { + MenuManager menuMgr = new MenuManager("#PopupMenu"); //$NON-NLS-1$ + menuMgr.setRemoveAllWhenShown(true); + menuMgr.addMenuListener(this::fillContextMenu); + queryString.setMenu(menuMgr.createContextMenu(queryString)); + } + + private void fillContextMenu(IMenuManager manager) { + updateTextActions(); + manager.add(executeQueryAction); + manager.add(explainQueryAction); + manager.add(new Separator()); + manager.add(undoAction); + manager.add(redoAction); + manager.add(new Separator()); + manager.add(textActions.get(ActionFactory.CUT.getId())); + manager.add(textActions.get(ActionFactory.COPY.getId())); + manager.add(textActions.get(ActionFactory.PASTE.getId())); + manager.add(new Separator()); + manager.add(textActions.get(ActionFactory.DELETE.getId())); + manager.add(textActions.get(ActionFactory.SELECT_ALL.getId())); + manager.add(new Separator()); + manager.add(commentLineAction); + manager.add(showReferenceAction); + } + + /** + * Line numbers in the editor's own colors: by default the column uses the system list colors, which do not follow + * the workbench theme (a black column with dark numbers in the dark theme). The colors are read on every paint, so + * they also pick up theme styling applied after the editor was created. + */ + private class ThemedLineNumberRulerColumn extends LineNumberRulerColumn { + private RGB lastForeground; + + @Override + protected Color getBackground(Display display) { + if (queryString == null || queryString.isDisposed()) { + return super.getBackground(display); + } + Color background = queryString.getBackground(); + // Numbers half way between text and background, like Eclipse's own line numbers + RGB fg = queryString.getForeground().getRGB(); + RGB bg = background.getRGB(); + RGB numbers = new RGB((fg.red + bg.red) / 2, (fg.green + bg.green) / 2, (fg.blue + bg.blue) / 2); + if (!numbers.equals(lastForeground)) { + lastForeground = numbers; + setForeground(new Color(display, numbers)); + } + return background; + } + } + + private class TextOperationAction extends Action { + private final int operation; + + TextOperationAction(int operation) { + this.operation = operation; + } + + void update() { + setEnabled(queryViewer.canDoOperation(operation)); + } + + @Override + public void run() { + queryViewer.doOperation(operation); + // The undo manager records that redo is possible only after the text change, so refresh again afterwards + updateTextActions(); + } + } + + /** + * Refreshes the enabled state of the text actions. It is stored rather than computed in isEnabled(), so the context + * menu and the key bindings (through the global action handlers) see the same up-to-date value. + */ + private void updateTextActions() { + undoAction.update(); + redoAction.update(); + for (TextOperationAction action : textActions.values()) { + action.update(); + } + } + + @Override + public void dispose() { + OPEN_PANES.remove(this); + // A query still running has no tab left to show its result in + Job.getJobManager().cancel(this); + // Closed while the editor had focus: focusLost may never come, which would leave the SQL key bindings active + // and the editor-wide Cut/Copy/Paste handlers pointing at this closed editor. Only handlers still ours are + // cleared, since another pane may have set its own since. + if (contextActivation != null) { + contextActivation.getContextService().deactivateContext(contextActivation); + contextActivation = null; + } + MultiPaneEditor editor = getEditor(); + if (editor != null && editor.getEditorSite() != null) { + IActionBars actionBars = editor.getEditorSite().getActionBars(); + boolean changed = false; + for (Map.Entry entry : textActions.entrySet()) { + if (actionBars.getGlobalActionHandler(entry.getKey()) == entry.getValue()) { + actionBars.setGlobalActionHandler(entry.getKey(), null); + changed = true; + } + } + if (changed) { + actionBars.updateActionBars(); + } + } + super.dispose(); + } + + /** + * The SQL pane of the editor that was opened or focused last, or null if the editor has none. + */ + static CalcitePane lastUsed(MultiPaneEditor editor) { + for (int i = OPEN_PANES.size() - 1; i >= 0; i--) { + CalcitePane pane = OPEN_PANES.get(i); + if (pane.getEditor() == editor && pane.queryString != null && !pane.queryString.isDisposed()) { + return pane; + } + } + return null; + } + + /** + * Adds a query at the end of the editor, or in place of the sample query of a new tab, and selects it so F5 runs + * just that query. Nothing is executed. + */ + void appendQuery(String sql) { + IDocument doc = queryViewer.getDocument(); + String text = doc.get(); + int start; + String insert; + if (text.equals(DEFAULT_QUERY)) { + start = HINT.length(); + insert = sql; + } else { + start = text.length(); + insert = (text.isEmpty() || text.endsWith("\n\n") ? "" : text.endsWith("\n") ? "\n" : "\n\n") + sql; + } + try { + doc.replace(start, text.length() - start, insert); + } catch (BadLocationException e) { + return; + } + int queryStart = start + insert.length() - sql.length(); + queryViewer.setSelectedRange(queryStart, sql.length()); + queryViewer.revealRange(queryStart, sql.length()); + queryString.setFocus(); + } + + private ISnapshot getSnapshot() { + return (ISnapshot) getEditor().getQueryContext().get(ISnapshot.class, null); + } + + /** + * Inserts text at the cursor, replacing the selection, as one undoable edit, and focuses the editor. + */ + private void insertText(String text) { + Point selection = queryViewer.getSelectedRange(); + try { + queryViewer.getDocument().replace(selection.x, selection.y, text); + } catch (BadLocationException e) { + return; + } + queryViewer.setSelectedRange(selection.x + text.length(), 0); + queryString.setFocus(); } public StyledText getQueryString() { return queryString; } - public String highlightError(Throwable e) { + /** + * Removes the error underline of an earlier run: it is drawn over the syntax colors, which repaint only text that + * is edited, so it would otherwise stay after the query is fixed or another one runs. + */ + public void clearErrorHighlight() { + if (!queryString.isDisposed()) { + queryViewer.invalidateTextPresentation(); + } + } + + /** + * Underlines the position of an SQL error and returns the error's message. + * + * @param sql the SQL that ran, which positions in the error refer to + * @param editorOffset where sql starts in the editor (a selection was run) + * @param prefixLength the length of text at the start of sql that is not in the editor (explain plan for) + */ + public String highlightError(Throwable e, String sql, int editorOffset, int prefixLength) { Throwable t; SqlParserPos errPos = null; for (t = e; t != null; t = t.getCause()) { @@ -245,10 +538,21 @@ public String highlightError(Throwable e) { return null; } - String sql = queryViewer.getDocument().get(); + clearErrorHighlight(); StyleRange style = new StyleRange(); - int start = SqlParserUtil.lineColToIndex(sql, errPos.getLineNum(), errPos.getColumnNum()); - int end = SqlParserUtil.lineColToIndex(sql, errPos.getEndLineNum(), errPos.getEndColumnNum()) + 1; + int start; + int end; + try { + start = SqlParserUtil.lineColToIndex(sql, errPos.getLineNum(), errPos.getColumnNum()); + end = SqlParserUtil.lineColToIndex(sql, errPos.getEndLineNum(), errPos.getEndColumnNum()) + 1; + } catch (RuntimeException outside) { + return t.getMessage(); // a position outside the SQL: nothing to underline + } + start = Math.max(start, prefixLength) - prefixLength + editorOffset; + end = Math.max(end, prefixLength) - prefixLength + editorOffset; + if (start < 0 || end <= start || end > queryString.getCharCount()) { + return t.getMessage(); // e.g. the text was edited while the query ran + } style.start = start; style.length = end - start; style.foreground = JFaceResources.getColorRegistry().get(JFacePreferences.ERROR_COLOR); @@ -258,24 +562,45 @@ public String highlightError(Throwable e) { return t.getMessage(); } - public void initQueryResult(QueryResult queryResult, PaneState state) { + /** + * Re-opens a query from the navigation history, or runs a query passed by another pane. + */ + @Override + public void initWithArgument(Object argument) { + if (argument instanceof String) { + queryViewer.getDocument().set((String) argument); + executeQueryAction.run(); + } else if (argument instanceof QueryResult) { + initQueryResult((QueryResult) argument, null, null); + } else if (argument instanceof PaneState) { + PaneState state = (PaneState) argument; + queryViewer.getDocument().set(state.getIdentifier()); + new ExecuteQueryAction(this, state, false).run(); + } + } + + /** + * Shows a query result and records it in the navigation history. + * + * @param sql the SQL that produced the result, used to re-run it from the history; the editor text when null + */ + public void initQueryResult(QueryResult queryResult, PaneState state, String sql) { IResult subject = queryResult.getSubject(); - // queryViewer.getDocument().set(subject.getOQLQuery()); + String identifier = sql == null ? queryString.getText() : sql.trim(); AbstractEditorPane pane = EditorPaneRegistry.instance().createNewPane( subject, this.getClass()); if (state == null) { for (PaneState child : getPaneState().getChildren()) { - if (queryString.getText().equals(child.getIdentifier())) { + if (identifier.equals(child.getIdentifier())) { state = child; break; } } if (state == null) { - state = new PaneState(PaneType.COMPOSITE_CHILD, getPaneState(), - queryString.getText(), true); + state = new PaneState(PaneType.COMPOSITE_CHILD, getPaneState(), identifier, true); state.setImage(getTitleImage()); } } @@ -285,10 +610,29 @@ public void initQueryResult(QueryResult queryResult, PaneState state) { createResultPane(pane, queryResult); } + /** + * Shows plain text (an error) in the result area, without adding it to the navigation history. + */ + public void showText(String text) { + AbstractEditorPane pane; + try { + pane = EditorPaneRegistry.instance().createNewPane("TextViewPane"); //$NON-NLS-1$ + } catch (Exception e) { + ErrorHelper.logThrowable(e); + return; + } + if (pane == null) { + return; + } + createResultPane(pane, text); + } + @Override public void contributeToToolBar(IToolBarManager manager) { manager.add(executeQueryAction); manager.add(explainQueryAction); + manager.add(showReferenceAction); + manager.add(toggleSchemaAction); super.contributeToToolBar(manager); } } diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteScanner.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteScanner.java index 898d323..520f4da 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteScanner.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteScanner.java @@ -5,16 +5,12 @@ import com.google.common.base.Splitter; import org.apache.calcite.jdbc.CalciteConnection; import org.apache.calcite.sql.parser.SqlAbstractParserImpl; -import org.eclipse.jface.text.TextAttribute; import org.eclipse.jface.text.rules.IRule; import org.eclipse.jface.text.rules.IWhitespaceDetector; import org.eclipse.jface.text.rules.IWordDetector; import org.eclipse.jface.text.rules.RuleBasedScanner; import org.eclipse.jface.text.rules.Token; import org.eclipse.jface.text.rules.WordRule; -import org.eclipse.swt.graphics.Color; -import org.eclipse.swt.graphics.RGB; -import org.eclipse.swt.widgets.Display; import java.sql.Connection; import java.sql.SQLException; @@ -45,10 +41,8 @@ public CalciteScanner() { List rules = new ArrayList<>(); Token other = new Token(CalcitePartitionScanner.OTHER); - Token keywordToken = new Token(new TextAttribute(new Color( - Display.getCurrent(), new RGB(127, 0, 85)))); - Token functionToken = new Token(new TextAttribute(new Color( - Display.getCurrent(), new RGB(26, 0, 192)))); + Token keywordToken = SqlColors.token(SqlColors.KEYWORD); + Token functionToken = SqlColors.token(SqlColors.FUNCTION); // "other" class is importantThis is required so Eclipse does not try to highlight in the middle of the words WordRule keywordRule = new WordRule(new CalciteWordDetector(), diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteSourceViewerConfiguration.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteSourceViewerConfiguration.java index 16345bf..1c21cf1 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteSourceViewerConfiguration.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CalciteSourceViewerConfiguration.java @@ -41,8 +41,9 @@ public IHyperlinkPresenter getHyperlinkPresenter(ISourceViewer sourceViewer) { public IContentAssistant getContentAssistant(ISourceViewer sourceViewer) { IContentAssistProcessor proc = new CalciteContentAssistantProcessor(); ContentAssistant assistant = new ContentAssistant(); - assistant.enableAutoActivation(false); - assistant.setAutoActivationDelay(500); + // Proposals pop up after '.' and '"' (see CalciteContentAssistantProcessor), as well as on Ctrl+Space + assistant.enableAutoActivation(true); + assistant.setAutoActivationDelay(300); assistant.setContentAssistProcessor(proc, IDocument.DEFAULT_CONTENT_TYPE); assistant.setContentAssistProcessor(proc, CalcitePartitionScanner.SQL_QUOTED_IDENTIFIER); return assistant; diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CommentScanner.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CommentScanner.java index 6e76054..e0396c0 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CommentScanner.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/CommentScanner.java @@ -1,14 +1,10 @@ package com.github.vlsi.mat.calcite.editor; -import org.eclipse.jface.text.TextAttribute; import org.eclipse.jface.text.rules.EndOfLineRule; import org.eclipse.jface.text.rules.IRule; import org.eclipse.jface.text.rules.MultiLineRule; import org.eclipse.jface.text.rules.RuleBasedScanner; import org.eclipse.jface.text.rules.Token; -import org.eclipse.swt.graphics.Color; -import org.eclipse.swt.graphics.RGB; -import org.eclipse.swt.widgets.Display; import java.util.ArrayList; import java.util.List; @@ -17,8 +13,7 @@ public class CommentScanner extends RuleBasedScanner { public CommentScanner() { List rules = new ArrayList<>(); - Token commentToken = new Token(new TextAttribute(new Color( - Display.getCurrent(), new RGB(63, 127, 95)))); + Token commentToken = SqlColors.token(SqlColors.COMMENT); rules.add(new EndOfLineRule("--", commentToken)); rules.add(new EndOfLineRule("//", commentToken)); rules.add(new MultiLineRule("/*", "*/", commentToken)); diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/OpenInSqlEditorQuery.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/OpenInSqlEditorQuery.java new file mode 100644 index 0000000..1b9a542 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/OpenInSqlEditorQuery.java @@ -0,0 +1,98 @@ +package com.github.vlsi.mat.calcite.editor; + +import com.github.vlsi.mat.calcite.SqlForObjects; + +import org.eclipse.mat.query.IContextObject; +import org.eclipse.mat.query.IQuery; +import org.eclipse.mat.query.IResult; +import org.eclipse.mat.query.annotations.Argument; +import org.eclipse.mat.query.annotations.Category; +import org.eclipse.mat.query.annotations.Help; +import org.eclipse.mat.query.annotations.Icon; +import org.eclipse.mat.query.annotations.Name; +import org.eclipse.mat.snapshot.ISnapshot; +import org.eclipse.mat.ui.editor.MultiPaneEditor; +import org.eclipse.mat.util.IProgressListener; +import org.eclipse.swt.widgets.Display; +import org.eclipse.ui.IEditorPart; +import org.eclipse.ui.IEditorReference; +import org.eclipse.ui.IWorkbenchPage; +import org.eclipse.ui.IWorkbenchWindow; +import org.eclipse.ui.PlatformUI; + +import java.util.List; + +/** + * "SQL / Open in SQL Editor" on the context menu of objects and classes in MAT's views: adds a query for the selection + * to the last used SQL tab (opening one if there is none) and selects it, without running it. Modelled on MAT's + * "Copy / OQL Query". + */ +@Name("Open in SQL Editor") +@Category("SQL") +@Icon("/resources/icons/plugin.png") +@Help("Adds a query that selects the chosen objects, or all instances of the chosen classes, to the SQL editor. " + + "Run it with F5.") +public class OpenInSqlEditorQuery implements IQuery { + @Argument + public ISnapshot snapshot; + + @Argument + public List elements; + + @Argument + public Display display; + + @Override + public IResult execute(IProgressListener listener) throws Exception { + String sql = SqlForObjects.build(snapshot, elements); + display.asyncExec(() -> { + MultiPaneEditor multiPaneEditor = editorOf(snapshot); + if (multiPaneEditor == null) { + return; + } + // Reuse the last SQL tab rather than opening one per query + CalcitePane pane = CalcitePane.lastUsed(multiPaneEditor); + if (pane == null) { + multiPaneEditor.addNewPage("CALCITE", null); + pane = CalcitePane.lastUsed(multiPaneEditor); + if (pane == null) { + return; + } + } else { + multiPaneEditor.bringPageToTop(pane.getPaneState()); + } + pane.appendQuery(sql); + }); + // The query is only added to the SQL editor, so there is no result to show here + throw new IProgressListener.OperationCanceledException(); + } + + /** + * The editor of the dump the objects belong to: addresses mean nothing in another dump. The active editor is + * usually that one, but not when the query was started from a view, or the user switched tabs meanwhile. + */ + private static MultiPaneEditor editorOf(ISnapshot snapshot) { + IWorkbenchWindow window = PlatformUI.getWorkbench().getActiveWorkbenchWindow(); + IWorkbenchPage page = window == null ? null : window.getActivePage(); + if (page == null) { + return null; + } + IEditorPart active = page.getActiveEditor(); + if (active instanceof MultiPaneEditor && snapshotOf((MultiPaneEditor) active) == snapshot) { + return (MultiPaneEditor) active; + } + for (IEditorReference reference : page.getEditorReferences()) { + IEditorPart editor = reference.getEditor(false); + if (editor instanceof MultiPaneEditor && snapshotOf((MultiPaneEditor) editor) == snapshot) { + page.activate(editor); + return (MultiPaneEditor) editor; + } + } + return null; + } + + private static ISnapshot snapshotOf(MultiPaneEditor editor) { + return editor.getQueryContext() == null ? null + : (ISnapshot) editor.getQueryContext().get(ISnapshot.class, null); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SchemaBrowser.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SchemaBrowser.java new file mode 100644 index 0000000..7c2630b --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SchemaBrowser.java @@ -0,0 +1,383 @@ +package com.github.vlsi.mat.calcite.editor; + +import com.github.vlsi.mat.calcite.CalciteDataSource; +import com.github.vlsi.mat.calcite.SchemaCatalog; + +import static com.github.vlsi.mat.calcite.SchemaCatalog.IDS_PREFIX; +import static com.github.vlsi.mat.calcite.SchemaCatalog.INSTANCEOF; +import static com.github.vlsi.mat.calcite.SchemaCatalog.NATIVE; +import static com.github.vlsi.mat.calcite.SqlIdentifiers.quote; +import static com.github.vlsi.mat.calcite.SqlIdentifiers.quotePath; + +import org.apache.calcite.jdbc.JavaTypeFactoryImpl; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.rel.type.RelDataTypeField; +import org.apache.calcite.schema.Schema; +import org.apache.calcite.schema.Table; +import org.eclipse.core.runtime.IProgressMonitor; +import org.eclipse.core.runtime.IStatus; +import org.eclipse.core.runtime.Status; +import org.eclipse.core.runtime.jobs.Job; +import org.eclipse.jface.resource.ImageDescriptor; +import org.eclipse.jface.resource.ImageRegistry; +import org.eclipse.jface.resource.JFaceResources; +import org.eclipse.jface.viewers.IStructuredSelection; +import org.eclipse.jface.viewers.ITreeContentProvider; +import org.eclipse.jface.viewers.LabelProvider; +import org.eclipse.jface.viewers.TreeViewer; +import org.eclipse.mat.snapshot.ISnapshot; +import org.eclipse.mat.snapshot.query.Icons; +import org.eclipse.swt.SWT; +import org.eclipse.swt.graphics.Image; +import org.eclipse.swt.layout.GridData; +import org.eclipse.swt.layout.GridLayout; +import org.eclipse.swt.widgets.Composite; +import org.eclipse.swt.widgets.Text; +import org.eclipse.ui.ISharedImages; +import org.eclipse.ui.PlatformUI; + +import java.net.URL; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.TreeSet; +import java.util.function.Consumer; + +/** + * Tree of the tables, columns and functions that SQL queries can use: classes by package, classes with their + * subclasses ({@code instanceof}), thread stacks and functions. Double-click inserts the name, quoted where SQL needs + * it. The filter box switches to a flat list of the classes whose name contains the text. + */ +class SchemaBrowser { + private static final int MAX_FILTER_RESULTS = 1000; + private static final int FILTER_DELAY_MILLIS = 200; + /** Shared, so ClassRowTypeCache (keyed by type factory) caches the row types across browsers. */ + private static final RelDataTypeFactory TYPE_FACTORY = new JavaTypeFactoryImpl(); + + private final Composite control; + private final Text filter; + private final TreeViewer viewer; + private final Consumer insert; + private Schema schema; + private List roots = Collections.emptyList(); + /** One instance, so a newer keystroke can cancel the refresh an older one scheduled. */ + private final Runnable refreshLater = this::refreshIfOpen; + + SchemaBrowser(Composite parent, ISnapshot snapshot, Consumer insert) { + this.insert = insert; + control = new Composite(parent, SWT.NONE); + GridLayout layout = new GridLayout(1, false); + layout.marginWidth = 0; + layout.marginHeight = 0; + layout.verticalSpacing = 2; + control.setLayout(layout); + + filter = new Text(control, SWT.SEARCH | SWT.ICON_SEARCH | SWT.ICON_CANCEL); + filter.setMessage("Filter tables"); + filter.setLayoutData(new GridData(SWT.FILL, SWT.CENTER, true, false)); + // Search once typing pauses, not on every keystroke + filter.addModifyListener(e -> { + filter.getDisplay().timerExec(-1, refreshLater); + filter.getDisplay().timerExec(FILTER_DELAY_MILLIS, refreshLater); + }); + + viewer = new TreeViewer(control, SWT.SINGLE | SWT.V_SCROLL | SWT.H_SCROLL); + viewer.getControl().setLayoutData(new GridData(SWT.FILL, SWT.FILL, true, true)); + viewer.setContentProvider(new ContentProvider()); + viewer.setLabelProvider(new NodeLabelProvider()); + viewer.addDoubleClickListener(e -> { + Object element = ((IStructuredSelection) e.getSelection()).getFirstElement(); + if (!(element instanceof Node)) { + return; + } + Node node = (Node) element; + if (node.insertText != null) { + this.insert.accept(node.insertText); + } else { + viewer.setExpandedState(node, !viewer.getExpandedState(node)); + } + }); + viewer.setInput(Collections.singletonList(new Node(Kind.MESSAGE, "Loading schema...", null))); + + loadSchema(snapshot); + } + + Composite getControl() { + return control; + } + + private void loadSchema(ISnapshot snapshot) { + if (snapshot == null) { + viewer.setInput(Collections.singletonList(new Node(Kind.MESSAGE, "No heap dump", null))); + return; + } + Job job = new Job("Loading SQL schema") { + @Override + protected IStatus run(IProgressMonitor monitor) { + Schema loaded; + try { + loaded = CalciteDataSource.getHeapSchema(snapshot); + } catch (Exception e) { + showMessage("Unable to load the schema: " + e.getMessage()); + return Status.OK_STATUS; + } + if (!control.isDisposed()) { + control.getDisplay().asyncExec(() -> { + if (control.isDisposed()) { + return; + } + schema = loaded; + roots = createRoots(loaded); + refreshInput(); + }); + } + return Status.OK_STATUS; + } + }; + job.setSystem(true); + job.schedule(); + } + + private void showMessage(String message) { + if (control.isDisposed()) { + return; + } + control.getDisplay().asyncExec(() -> { + if (!control.isDisposed()) { + viewer.setInput(Collections.singletonList(new Node(Kind.MESSAGE, message, null))); + } + }); + } + + private void refreshIfOpen() { + if (!filter.isDisposed()) { + refreshInput(); + } + } + + private void refreshInput() { + if (schema == null) { + return; + } + String text = filter.getText().trim(); + viewer.getTree().setRedraw(false); + try { + viewer.setInput(text.isEmpty() ? roots : filterClasses(text)); + } finally { + viewer.getTree().setRedraw(true); + } + } + + private static List createRoots(Schema schema) { + List roots = new ArrayList<>(); + Node classes = new Node(Kind.GROUP, "Classes", null); + classes.schema = schema; + roots.add(classes); + Schema instanceOf = schema.getSubSchema(INSTANCEOF); + if (instanceOf != null) { + Node node = new Node(Kind.GROUP, "Classes and subclasses (instanceof)", null); + node.schema = instanceOf; + node.path = INSTANCEOF; + roots.add(node); + } + Schema nativeSchema = schema.getSubSchema(NATIVE); + if (nativeSchema != null) { + Node node = new Node(Kind.GROUP, "Thread stacks (native)", null); + node.schema = nativeSchema; + node.path = NATIVE; + roots.add(node); + } + Node functions = new Node(Kind.FUNCTIONS, "Functions", null); + functions.schema = schema; + roots.add(functions); + return roots; + } + + /** Tables whose name contains the text, as a flat list. */ + private List filterClasses(String text) { + List result = new ArrayList<>(); + for (SchemaCatalog.TableRef table : SchemaCatalog.findTables(schema, text, MAX_FILTER_RESULTS + 1)) { + if (result.size() == MAX_FILTER_RESULTS) { + result.add(new Node(Kind.MESSAGE, "More than " + MAX_FILTER_RESULTS + " tables, refine the filter", null)); + break; + } + result.add(classNode(table.schema, table.tableName, table.fullName, table.sqlName)); + } + if (result.isEmpty()) { + result.add(new Node(Kind.MESSAGE, "No table matches", null)); + } + return result; + } + + private static Node classNode(Schema owner, String tableName, String label, String insertText) { + Node node = new Node(Kind.CLASS, label, insertText); + node.schema = owner; + node.tableName = tableName; + return node; + } + + /** Children of a package-like schema: sub-packages, then classes. */ + private static List schemaChildren(Node parent) { + List children = new ArrayList<>(); + boolean root = parent.path == null; + for (String name : new TreeSet<>(parent.schema.getSubSchemaNames())) { + if (root && (INSTANCEOF.equals(name) || NATIVE.equals(name))) { + continue; + } + Node node = new Node(Kind.PACKAGE, name, null); + node.schema = parent.schema.getSubSchema(name); + node.path = root ? name : parent.path + "." + name; + children.add(node); + } + for (String name : new TreeSet<>(parent.schema.getTableNames())) { + if (name.startsWith(IDS_PREFIX)) { + continue; + } + if (name.contains(".")) { + // The root and instanceof schemas also list every class by its full name; the package tree shows those + continue; + } + // Schema path form, e.g. java.util."HashMap$Node", instanceof.java.util.AbstractMap or native.ThreadStackFrames + String insertText = root ? quote(name) : quotePath(parent.path) + "." + quote(name); + children.add(classNode(parent.schema, name, name, insertText)); + } + return children; + } + + private static List columns(Node classNode) { + Table table = classNode.schema.getTable(classNode.tableName); + if (table == null) { + return Collections.emptyList(); + } + List children = new ArrayList<>(); + try { + for (RelDataTypeField field : table.getRowType(TYPE_FACTORY).getFieldList()) { + Node node = new Node(Kind.COLUMN, field.getName() + " : " + field.getType().getSqlTypeName(), + quote(field.getName())); + children.add(node); + } + // Heap object tables (first column "this") also have sizes, computed through the object + if (!children.isEmpty() && "this".equals(table.getRowType(TYPE_FACTORY).getFieldList().get(0).getName())) { + children.add(new Node(Kind.COLUMN, "this['@shallow'] : shallow heap size", "this['@shallow']")); + children.add(new Node(Kind.COLUMN, "this['@retained'] : retained heap size", "this['@retained']")); + } + } catch (RuntimeException e) { + children.add(new Node(Kind.MESSAGE, "Unable to read columns: " + e.getMessage(), null)); + } + return children; + } + + private static List functions(Node functionsNode) { + List children = new ArrayList<>(); + for (String name : SchemaCatalog.functionNames(functionsNode.schema)) { + children.add(new Node(Kind.FUNCTION, name, name + "(")); + } + return children; + } + + private enum Kind { + GROUP, PACKAGE, CLASS, COLUMN, FUNCTIONS, FUNCTION, MESSAGE + } + + private static final class Node { + final Kind kind; + final String label; + /** Text inserted on double-click; null for nodes that only expand. */ + final String insertText; + Schema schema; + /** Dotted package path of package-like nodes, null for the root schema. */ + String path; + String tableName; + List children; + + Node(Kind kind, String label, String insertText) { + this.kind = kind; + this.label = label; + this.insertText = insertText; + } + + List children() { + if (children == null) { + switch (kind) { + case GROUP: + case PACKAGE: + children = schemaChildren(this); + break; + case CLASS: + children = columns(this); + break; + case FUNCTIONS: + children = functions(this); + break; + default: + children = Collections.emptyList(); + break; + } + } + return children; + } + + boolean hasChildren() { + return kind != Kind.COLUMN && kind != Kind.FUNCTION && kind != Kind.MESSAGE; + } + } + + private static final class ContentProvider implements ITreeContentProvider { + @Override + public Object[] getElements(Object input) { + return ((List) input).toArray(); + } + + @Override + public Object[] getChildren(Object parent) { + return ((Node) parent).children().toArray(); + } + + @Override + public Object getParent(Object element) { + return null; + } + + @Override + public boolean hasChildren(Object element) { + return ((Node) element).hasChildren(); + } + } + + private static final class NodeLabelProvider extends LabelProvider { + @Override + public String getText(Object element) { + return ((Node) element).label; + } + + @Override + public Image getImage(Object element) { + ISharedImages shared = PlatformUI.getWorkbench().getSharedImages(); + switch (((Node) element).kind) { + case GROUP: + case PACKAGE: + case FUNCTIONS: + return shared.getImage(ISharedImages.IMG_OBJ_FOLDER); + case CLASS: + return image(Icons.CLASS); + case COLUMN: + case FUNCTION: + return shared.getImage(ISharedImages.IMG_OBJ_ELEMENT); + default: + return null; + } + } + + /** Images from the shared JFace registry, which owns and disposes them. */ + private static Image image(URL url) { + ImageRegistry registry = JFaceResources.getImageRegistry(); + String key = url.toExternalForm(); + Image image = registry.get(key); + if (image == null) { + registry.put(key, ImageDescriptor.createFromURL(url)); + image = registry.get(key); + } + return image; + } + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlColors.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlColors.java new file mode 100644 index 0000000..ecb783e --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlColors.java @@ -0,0 +1,29 @@ +package com.github.vlsi.mat.calcite.editor; + +import org.eclipse.jface.text.TextAttribute; +import org.eclipse.jface.text.rules.Token; +import org.eclipse.swt.graphics.Color; +import org.eclipse.ui.PlatformUI; + +/** + * Syntax colors of the SQL editor. The values are defined as theme colors in plugin.xml, so they follow the + * current workbench theme and can be edited in Preferences / Appearance / Colors and Fonts. + * The registry owns the colors, so they must not be disposed. + */ +public final class SqlColors { + public static final String KEYWORD = "com.github.vlsi.mat.calcite.sql_keyword"; + public static final String FUNCTION = "com.github.vlsi.mat.calcite.sql_function"; + public static final String COMMENT = "com.github.vlsi.mat.calcite.sql_comment"; + public static final String STRING = "com.github.vlsi.mat.calcite.sql_string"; + + private SqlColors() { + } + + public static Color get(String id) { + return PlatformUI.getWorkbench().getThemeManager().getCurrentTheme().getColorRegistry().get(id); + } + + public static Token token(String id) { + return new Token(new TextAttribute(get(id))); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlEditorCommandHandler.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlEditorCommandHandler.java new file mode 100644 index 0000000..6ee2340 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlEditorCommandHandler.java @@ -0,0 +1,37 @@ +package com.github.vlsi.mat.calcite.editor; + +import org.eclipse.core.commands.AbstractHandler; +import org.eclipse.core.commands.ExecutionEvent; +import org.eclipse.mat.ui.editor.AbstractEditorPane; +import org.eclipse.mat.ui.editor.MultiPaneEditor; +import org.eclipse.ui.IEditorPart; +import org.eclipse.ui.IWorkbenchPage; +import org.eclipse.ui.IWorkbenchWindow; +import org.eclipse.ui.PlatformUI; + +/** + * Handles the SQL editor commands declared in plugin.xml. Their key bindings live in the + * {@link CalcitePane#CONTEXT_ID} context, a child of MAT's editor context, so they take precedence over MAT's OQL + * bindings for the same keys while the SQL editor has focus. + */ +public class SqlEditorCommandHandler extends AbstractHandler { + @Override + public Object execute(ExecutionEvent event) { + CalcitePane pane = activeCalcitePane(); + if (pane != null) { + pane.runCommand(event.getCommand().getId()); + } + return null; + } + + private static CalcitePane activeCalcitePane() { + IWorkbenchWindow window = PlatformUI.getWorkbench().getActiveWorkbenchWindow(); + IWorkbenchPage page = window == null ? null : window.getActivePage(); + IEditorPart editor = page == null ? null : page.getActiveEditor(); + if (!(editor instanceof MultiPaneEditor)) { + return null; + } + AbstractEditorPane pane = ((MultiPaneEditor) editor).getActiveEditor(); + return pane instanceof CalcitePane ? (CalcitePane) pane : null; + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlReferencePane.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlReferencePane.java new file mode 100644 index 0000000..e0dd49e --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/SqlReferencePane.java @@ -0,0 +1,46 @@ +package com.github.vlsi.mat.calcite.editor; + +import com.github.vlsi.mat.calcite.SqlReference; + +import org.eclipse.jface.resource.JFaceResources; +import org.eclipse.mat.ui.editor.AbstractEditorPane; +import org.eclipse.swt.SWT; +import org.eclipse.swt.custom.StyledText; +import org.eclipse.swt.graphics.Image; +import org.eclipse.swt.widgets.Composite; +import org.eclipse.ui.ISharedImages; +import org.eclipse.ui.PlatformUI; + +/** + * Read-only SQL quick reference, opened as its own editor tab so it does not replace the query results. + */ +public class SqlReferencePane extends AbstractEditorPane { + /** Pane id, see plugin.xml. */ + public static final String ID = "CALCITE_REFERENCE"; + + private StyledText text; + + @Override + public void createPartControl(Composite parent) { + text = new StyledText(parent, SWT.MULTI | SWT.READ_ONLY | SWT.H_SCROLL | SWT.V_SCROLL); + text.setFont(JFaceResources.getFont(JFaceResources.TEXT_FONT)); + text.setMargins(8, 8, 8, 8); + text.setText(SqlReference.TEXT); + } + + @Override + public void setFocus() { + text.setFocus(); + } + + @Override + public String getTitle() { + return "SQL Reference"; + } + + @Override + public Image getTitleImage() { + // Shared workbench image, must not be disposed + return PlatformUI.getWorkbench().getSharedImages().getImage(ISharedImages.IMG_LCL_LINKTO_HELP); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/StringScanner.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/StringScanner.java index 2b0e4dc..fb500be 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/StringScanner.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/StringScanner.java @@ -1,13 +1,9 @@ package com.github.vlsi.mat.calcite.editor; -import org.eclipse.jface.text.TextAttribute; import org.eclipse.jface.text.rules.IRule; import org.eclipse.jface.text.rules.RuleBasedScanner; import org.eclipse.jface.text.rules.SingleLineRule; import org.eclipse.jface.text.rules.Token; -import org.eclipse.swt.graphics.Color; -import org.eclipse.swt.graphics.RGB; -import org.eclipse.swt.widgets.Display; import java.util.ArrayList; import java.util.List; @@ -16,8 +12,7 @@ public class StringScanner extends RuleBasedScanner { public StringScanner() { List rules = new ArrayList<>(); - Token stringToken = new Token(new TextAttribute(new Color( - Display.getCurrent(), new RGB(42, 0, 255)))); + Token stringToken = SqlColors.token(SqlColors.STRING); rules.add(new SingleLineRule("'", "'", stringToken)); rules.add(new SingleLineRule("\"", "\"", stringToken)); diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/TransientResizeScrollFix.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/TransientResizeScrollFix.java new file mode 100644 index 0000000..6fc0fc7 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/editor/TransientResizeScrollFix.java @@ -0,0 +1,75 @@ +package com.github.vlsi.mat.calcite.editor; + +import org.eclipse.swt.SWT; +import org.eclipse.swt.custom.StyledText; +import org.eclipse.swt.widgets.Display; +import org.eclipse.swt.widgets.Event; +import org.eclipse.swt.widgets.Listener; + +/** + * Keeps the query editor's scroll position across a transient resize. + * + *

When a query result adds buttons to MAT's toolbar, JFace's {@code ToolBarManager.relayout} packs the toolbar's + * parent, which briefly lays the whole editor area out at its preferred size (as wide as the longest SQL line) before + * {@code requestLayout} restores it. While the editor is taller, {@code StyledText} scrolls up to fill the space below + * the last line ("claim bottom free space"), and it does not scroll back when the editor shrinks again. With the + * editor scrolled to the end, running a query therefore made the text jump. + * + *

The fix records the scroll position just before a resize (a display filter runs before StyledText's own resize + * handling). If the editor grows, scrolls up, and returns to its original height shortly after, that position is + * restored. Lasting resizes, such as dragging the sash, are left alone. + */ +final class TransientResizeScrollFix { + /** How long a resize may last to count as transient. */ + private static final int TRANSIENT_MILLIS = 500; + + private final StyledText text; + private int lastHeight; + private int topBeforeResize; + /** Height and scroll position to return to when the editor shrinks back, or -1 when nothing is pending. */ + private int restoreHeight = -1; + private int restoreTop; + private long restoreUntil; + + private TransientResizeScrollFix(StyledText text) { + this.text = text; + this.lastHeight = text.getClientArea().height; + } + + static void install(StyledText text) { + TransientResizeScrollFix fix = new TransientResizeScrollFix(text); + Display display = text.getDisplay(); + Listener before = fix::beforeResize; + display.addFilter(SWT.Resize, before); + text.addListener(SWT.Resize, fix::afterResize); + text.addDisposeListener(e -> display.removeFilter(SWT.Resize, before)); + } + + private void beforeResize(Event e) { + if (e.widget == text) { + topBeforeResize = text.getTopPixel(); + } + } + + private void afterResize(Event e) { + int height = text.getClientArea().height; + int top = text.getTopPixel(); + long now = System.currentTimeMillis(); + if (height > lastHeight && top < topBeforeResize) { + // Grew and scrolled up to fill the space below the last line + if (restoreHeight < 0 || now > restoreUntil) { + restoreHeight = lastHeight; + restoreTop = topBeforeResize; + } + restoreUntil = now + TRANSIENT_MILLIS; + } else if (restoreHeight >= 0) { + if (height == restoreHeight && now <= restoreUntil) { + text.setTopPixel(restoreTop); + } + if (height <= restoreHeight || now > restoreUntil) { + restoreHeight = -1; + } + } + lastHeight = height; + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/DateFunctions.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/DateFunctions.java new file mode 100644 index 0000000..b7c0123 --- /dev/null +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/functions/DateFunctions.java @@ -0,0 +1,431 @@ +package com.github.vlsi.mat.calcite.functions; + +import com.github.vlsi.mat.calcite.HeapReference; + +import org.eclipse.mat.SnapshotException; +import org.eclipse.mat.snapshot.ISnapshot; +import org.eclipse.mat.snapshot.model.Field; +import org.eclipse.mat.snapshot.model.ObjectReference; +import org.eclipse.mat.snapshot.model.IClass; +import org.eclipse.mat.snapshot.model.IObject; + +import java.time.DateTimeException; +import java.time.Instant; +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.time.ZonedDateTime; +import java.time.format.DateTimeFormatter; +import java.util.Collections; +import java.util.HashSet; +import java.util.Map; +import java.util.Optional; +import java.util.WeakHashMap; +import java.util.Set; + +/** + * SQL functions that turn date and time objects of the heap into readable text and epoch numbers. Supported: + * java.util.Date (and java.sql.Date, Time, Timestamp), java.util.Calendar, java.time.Instant, LocalDate, LocalTime, + * LocalDateTime, ZonedDateTime, OffsetDateTime and Duration, and plain numbers as epoch milliseconds. + * + *

Values that are a point in time without a zone of their own (Date, Instant, epoch numbers) are shown in UTC + * unless a zone is given, e.g. toDateTime(this, 'Europe/Madrid'). Calendar, ZonedDateTime and OffsetDateTime are + * shown in their own zone or offset. Local values have no zone and are shown as they are. + */ +public class DateFunctions extends HeapFunctionsBase { + private static final DateTimeFormatter DATE_TIME = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.SSS"); + private static final DateTimeFormatter DATE = DateTimeFormatter.ofPattern("yyyy-MM-dd"); + private static final DateTimeFormatter TIME = DateTimeFormatter.ofPattern("HH:mm:ss.SSS"); + + // ---------------------------------------------------------------- SQL functions + + /** Readable date and time of a heap date object or epoch milliseconds, e.g. 2026-10-01 11:11:11.224. */ + @SuppressWarnings("unused") + public static String toDateTime(Object r) { + return toDateTime(r, null); + } + + /** Same as toDateTime(r), with points in time shown in the given zone, e.g. 'Europe/Madrid' or '+02:00'. */ + @SuppressWarnings("unused") + public static String toDateTime(Object r, String zone) { + Value value = read(r); + return value == null ? null : value.format(zone(zone, r)); + } + + /** Readable date and time of epoch milliseconds, in UTC. */ + @SuppressWarnings("unused") + public static String millisToDateTime(long millis) { + return millisToDateTime(millis, null); + } + + @SuppressWarnings("unused") + public static String millisToDateTime(long millis, String zone) { + return Value.instant(Instant.ofEpochMilli(millis), Kind.DATE_TIME).format(zone(zone, null)); + } + + /** Readable date and time of epoch seconds, in UTC. */ + @SuppressWarnings("unused") + public static String secondsToDateTime(long seconds) { + return secondsToDateTime(seconds, null); + } + + @SuppressWarnings("unused") + public static String secondsToDateTime(long seconds, String zone) { + return Value.instant(Instant.ofEpochSecond(seconds), Kind.DATE_TIME).format(zone(zone, null)); + } + + /** + * Epoch milliseconds of a heap date object. Local date-times and dates (no zone of their own) are taken as UTC; a + * LocalTime has no epoch value and gives null. + */ + @SuppressWarnings("unused") + public static Long toEpochMillis(Object r) { + return toEpochMillis(r, null); + } + + /** Same as toEpochMillis(r), with local date-times and dates taken in the given zone. */ + @SuppressWarnings("unused") + public static Long toEpochMillis(Object r, String zone) { + Value value = read(r); + return value == null ? null : value.epochMillis(zone(zone, r)); + } + + /** A java.time.Duration, or a number of milliseconds, as readable text, e.g. 1d 2h 3m 4.567s. */ + @SuppressWarnings("unused") + public static String toDuration(Object r) { + long seconds; + long nanos; + if (r instanceof Number) { + long millis = ((Number) r).longValue(); + seconds = Math.floorDiv(millis, 1000); + nanos = Math.floorMod(millis, 1000) * 1_000_000L; + } else { + IObject o = heapObject(r); + if (o == null || !isA(o, "java.time.Duration")) { + return null; + } + try { + seconds = number(o, "seconds"); + nanos = number(o, "nanos"); + } catch (SnapshotException e) { + return null; + } + } + return formatDuration(seconds, nanos); + } + + // ---------------------------------------------------------------- reading heap objects + + private enum Kind { + DATE_TIME, DATE, TIME + } + + /** A date or time read from the heap. */ + private static final class Value { + final Kind kind; + /** Point in time, or null for local values. */ + final Instant instant; + /** Zone of its own (Calendar, ZonedDateTime, OffsetDateTime), or null. */ + final ZoneId ownZone; + /** Local value, when there is no point in time. */ + final LocalDateTime local; + /** Zone to show the point in time in by default, e.g. the dumped JVM's zone for java.sql.Date; null for UTC. */ + final ZoneId defaultZone; + + private Value(Kind kind, Instant instant, ZoneId ownZone, LocalDateTime local, ZoneId defaultZone) { + this.kind = kind; + this.instant = instant; + this.ownZone = ownZone; + this.local = local; + this.defaultZone = defaultZone; + } + + static Value instant(Instant instant, Kind kind) { + return new Value(kind, instant, null, null, null); + } + + /** A point in time the application meant in its JVM's default zone, such as java.sql.Date. */ + static Value inJvmZone(Instant instant, Kind kind, ZoneId jvmZone) { + return new Value(kind, instant, null, null, jvmZone); + } + + static Value zoned(Instant instant, ZoneId zone) { + return new Value(Kind.DATE_TIME, instant, zone, null, null); + } + + static Value local(LocalDateTime local, Kind kind) { + return new Value(kind, null, null, local, null); + } + + String format(ZoneId requested) { + if (instant == null) { + return pattern().format(local); + } + if (ownZone != null && requested == null) { + // In the value's own zone, which the text names, e.g. 2026-10-01 11:11:11.224 +02:00 + return pattern().format(ZonedDateTime.ofInstant(instant, ownZone)) + " " + ownZone.getId(); + } + ZoneId zone = requested != null ? requested : defaultZone != null ? defaultZone : ZoneOffset.UTC; + return pattern().format(ZonedDateTime.ofInstant(instant, zone)); + } + + Long epochMillis(ZoneId requested) { + if (instant != null) { + return instant.toEpochMilli(); + } + if (kind == Kind.TIME) { + return null; + } + return local.atZone(requested == null ? ZoneOffset.UTC : requested).toInstant().toEpochMilli(); + } + + private DateTimeFormatter pattern() { + return kind == Kind.DATE ? DATE : kind == Kind.TIME ? TIME : DATE_TIME; + } + } + + private static Value read(Object r) { + if (r instanceof Number) { + return Value.instant(Instant.ofEpochMilli(((Number) r).longValue()), Kind.DATE_TIME); + } + IObject o = heapObject(r); + if (o == null) { + return null; + } + try { + Set types = types(o); + if (types.contains("java.util.Date") && !dateTimeIsCurrent(o)) { + return null; // changed through a deprecated setter: fastTime is stale until the JVM normalizes it + } + if (types.contains("java.sql.Timestamp")) { + // Whole seconds in fastTime, the fraction in nanos + long seconds = Math.floorDiv(number(o, "fastTime"), 1000); + return Value.instant(Instant.ofEpochSecond(seconds, number(o, "nanos")), Kind.DATE_TIME); + } + // java.sql.Date and Time are built in the JVM's default zone (Date.valueOf("2026-10-01") is local midnight), + // so they read right only in that zone + if (types.contains("java.sql.Date")) { + return Value.inJvmZone(Instant.ofEpochMilli(number(o, "fastTime")), Kind.DATE, jvmZone(o.getSnapshot())); + } + if (types.contains("java.sql.Time")) { + return Value.inJvmZone(Instant.ofEpochMilli(number(o, "fastTime")), Kind.TIME, jvmZone(o.getSnapshot())); + } + if (types.contains("java.util.Date")) { + return Value.instant(Instant.ofEpochMilli(number(o, "fastTime")), Kind.DATE_TIME); + } + if (types.contains("java.util.Calendar")) { + if (!Boolean.TRUE.equals(o.resolveValue("isTimeSet"))) { + return null; // fields set since the time was computed: "time" is stale until the JVM recomputes it + } + Instant instant = Instant.ofEpochMilli(number(o, "time")); + ZoneId zone = timeZone(o.resolveValue("zone")); + return zone == null ? Value.instant(instant, Kind.DATE_TIME) : Value.zoned(instant, zone); + } + if (types.contains("java.time.Instant")) { + return Value.instant(Instant.ofEpochSecond(number(o, "seconds"), number(o, "nanos")), Kind.DATE_TIME); + } + if (types.contains("java.time.LocalDateTime")) { + LocalDateTime local = localDateTime(o); + return local == null ? null : Value.local(local, Kind.DATE_TIME); + } + if (types.contains("java.time.LocalDate")) { + return Value.local(localDate(o).atStartOfDay(), Kind.DATE); + } + if (types.contains("java.time.LocalTime")) { + return Value.local(LocalDate.EPOCH.atTime(localTime(o)), Kind.TIME); + } + if (types.contains("java.time.ZonedDateTime") || types.contains("java.time.OffsetDateTime")) { + LocalDateTime local = localDateTime((IObject) o.resolveValue("dateTime")); + ZoneOffset offset = offset(o.resolveValue("offset")); + if (local == null || offset == null) { + return null; + } + ZoneId zone = types.contains("java.time.ZonedDateTime") ? zoneId(o.resolveValue("zone"), offset) : offset; + return Value.zoned(local.toInstant(offset), zone); + } + return null; + } catch (SnapshotException | DateTimeException | ClassCastException e) { + return null; // not a value this dump lets us read + } + } + + private static LocalDateTime localDateTime(IObject o) throws SnapshotException { + if (o == null) { + return null; + } + IObject date = (IObject) o.resolveValue("date"); + IObject time = (IObject) o.resolveValue("time"); + return date == null || time == null ? null : LocalDateTime.of(localDate(date), localTime(time)); + } + + private static LocalDate localDate(IObject o) throws SnapshotException { + return LocalDate.of((int) number(o, "year"), (int) number(o, "month"), (int) number(o, "day")); + } + + private static LocalTime localTime(IObject o) throws SnapshotException { + return LocalTime.of((int) number(o, "hour"), (int) number(o, "minute"), (int) number(o, "second"), + (int) number(o, "nano")); + } + + private static ZoneOffset offset(Object zoneOffset) throws SnapshotException { + return zoneOffset instanceof IObject + ? ZoneOffset.ofTotalSeconds((int) number((IObject) zoneOffset, "totalSeconds")) : null; + } + + /** The zone of a ZonedDateTime: a region such as Europe/Madrid, or the offset itself. */ + private static ZoneId zoneId(Object zone, ZoneOffset offset) throws SnapshotException { + if (zone instanceof IObject && types((IObject) zone).contains("java.time.ZoneRegion")) { + String id = string(((IObject) zone).resolveValue("id")); + if (id != null) { + try { + return ZoneId.of(id); + } catch (DateTimeException e) { + // a zone id this JVM does not know: fall back to the offset + } + } + } + return offset; + } + + /** The zone of a java.util.TimeZone (e.g. sun.util.calendar.ZoneInfo), from its ID. */ + private static ZoneId timeZone(Object timeZone) throws SnapshotException { + if (!(timeZone instanceof IObject)) { + return null; + } + String id = string(((IObject) timeZone).resolveValue("ID")); + if (id == null) { + return null; + } + try { + return ZoneId.of(id, ZoneId.SHORT_IDS); + } catch (DateTimeException e) { + return null; + } + } + + /** Default zone of the dumped JVM, from java.util.TimeZone.defaultTimeZone, per snapshot. */ + private static final Map> JVM_ZONES = + Collections.synchronizedMap(new WeakHashMap<>()); + + private static ZoneId jvmZone(ISnapshot snapshot) { + return JVM_ZONES.computeIfAbsent(snapshot, s -> { + try { + for (IClass timeZoneClass : s.getClassesByName("java.util.TimeZone", false)) { + for (Field field : timeZoneClass.getStaticFields()) { + if ("defaultTimeZone".equals(field.getName()) && field.getValue() instanceof ObjectReference) { + return Optional.ofNullable(timeZone(((ObjectReference) field.getValue()).getObject())); + } + } + } + } catch (SnapshotException e) { + // fall through: not known + } + return Optional.empty(); + }).orElse(null); + } + + // ---------------------------------------------------------------- helpers + + private static IObject heapObject(Object r) { + HeapReference ref = ensureHeapReference(r); + return ref == null ? null : ref.getIObject(); + } + + /** The class of an object and its superclasses. */ + private static Set types(IObject o) { + Set names = new HashSet<>(); + for (IClass c = o.getClazz(); c != null; c = c.getSuperClass()) { + names.add(c.getName()); + } + return names; + } + + private static boolean isA(IObject o, String className) { + return types(o).contains(className); + } + + private static long number(IObject o, String field) throws SnapshotException { + Object value = o.resolveValue(field); + if (!(value instanceof Number)) { + throw new SnapshotException("No numeric field " + field + " in " + o.getTechnicalName()); + } + return ((Number) value).longValue(); + } + + private static String string(Object value) { + return value instanceof IObject ? ((IObject) value).getClassSpecificName() : null; + } + + /** + * The zone argument: a zone id or offset, or 'jvm' for the default zone of the JVM the heap dump was taken from. + */ + private static ZoneId zone(String zone, Object heapValue) { + if (zone == null || zone.trim().isEmpty()) { + return null; + } + if ("jvm".equalsIgnoreCase(zone.trim())) { + IObject o = heapObject(heapValue); + ZoneId jvm = o == null ? null : jvmZone(o.getSnapshot()); + if (jvm == null) { + throw new IllegalArgumentException("The dumped JVM's time zone is not known" + + (o == null ? " for a plain number; pass a zone id instead" : " in this heap dump")); + } + return jvm; + } + try { + return ZoneId.of(zone.trim(), ZoneId.SHORT_IDS); + } catch (DateTimeException e) { + throw new IllegalArgumentException("Unknown time zone: " + zone + + " (use e.g. 'UTC', 'Europe/Madrid', '+02:00', or 'jvm' for the dumped JVM's zone)"); + } + } + + /** + * Formats seconds plus nanoseconds (0 to 999,999,999, as java.time.Duration stores them) without overflow: + * Duration.ofSeconds(Long.MAX_VALUE), a common "forever", is about 292 billion years. + */ + /** + * Whether a java.util.Date's fastTime is its time. A deprecated setter (setHours...) changes only the cached + * calendar fields (cdate) and marks them not normalized; fastTime follows when the JVM next reads the time. + */ + private static boolean dateTimeIsCurrent(IObject date) throws SnapshotException { + Object cdate = date.resolveValue("cdate"); + return !(cdate instanceof IObject) || !Boolean.FALSE.equals(((IObject) cdate).resolveValue("normalized")); + } + + static String formatDuration(long seconds, long nanos) { + boolean negative = seconds < 0; + long absSeconds = seconds; + long absNanos = nanos; + if (negative) { + if (absNanos > 0) { + absSeconds++; // -1.25 s is stored as -2 s + 0.75 s + absNanos = 1_000_000_000L - absNanos; + } + absSeconds = -absSeconds; // for Long.MIN_VALUE this stays 2^63, which the unsigned arithmetic below reads right + } + long millis = absNanos / 1_000_000L; + long days = Long.divideUnsigned(absSeconds, 86_400); + long rest = Long.remainderUnsigned(absSeconds, 86_400); + long hours = rest / 3600; + long minutes = rest / 60 % 60; + long secs = rest % 60; + StringBuilder sb = new StringBuilder(negative ? "-" : ""); + if (days > 0) { + sb.append(Long.toUnsignedString(days)).append("d "); + } + if (days > 0 || hours > 0) { + sb.append(hours).append("h "); + } + if (days > 0 || hours > 0 || minutes > 0) { + sb.append(minutes).append("m "); + } + sb.append(secs); + if (millis > 0) { + sb.append('.').append(String.format(java.util.Locale.ROOT, "%03d", millis)); + } + return sb.append('s').toString(); + } +} diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/neo/PackageSchema.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/neo/PackageSchema.java index 8d12782..db79281 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/neo/PackageSchema.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/neo/PackageSchema.java @@ -1,6 +1,7 @@ package com.github.vlsi.mat.calcite.neo; import com.github.vlsi.mat.calcite.functions.CollectionsFunctions; +import com.github.vlsi.mat.calcite.functions.DateFunctions; import com.github.vlsi.mat.calcite.functions.HeapFunctions; import com.github.vlsi.mat.calcite.functions.SnapshotFunctions; import com.github.vlsi.mat.calcite.functions.TableFunctions; @@ -101,6 +102,7 @@ public static PackageSchema resolveSchema(ISnapshot snapshot) { // Create functions for schema ImmutableMultimap.Builder builder = ImmutableMultimap.builder(); builder.putAll(ScalarFunctionImpl.functions(HeapFunctions.class)); + builder.putAll(ScalarFunctionImpl.functions(DateFunctions.class)); builder.putAll(CollectionsFunctions.createAll()); builder.putAll(TableFunctions.createAll()); builder.putAll(SnapshotFunctions.createAll(snapshot)); @@ -138,6 +140,9 @@ public static PackageSchema resolveSchema(ISnapshot snapshot) { // Make class available via "instanceof.package.name.ClassName" defaultSchema.addClass("instanceof." + fullClassName, withSubClasses); + // Make class available via instanceof."package.name.ClassName", like "package.name.ClassName" in the root + instanceOfPackage.addClass(fullClassName, withSubClasses); + // Make class available via instanceof.package.name.ClassName PackageSchema instanceOfSchema = getPackage(instanceOfPackage, fullClassName); instanceOfSchema.addClass(simpleClassName, withSubClasses); diff --git a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/schema/objects/InstanceIdsByClassTable.java b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/schema/objects/InstanceIdsByClassTable.java index f3a0982..1dda812 100644 --- a/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/schema/objects/InstanceIdsByClassTable.java +++ b/MatCalcitePlugin/src/com/github/vlsi/mat/calcite/schema/objects/InstanceIdsByClassTable.java @@ -1,5 +1,7 @@ package com.github.vlsi.mat.calcite.schema.objects; +import com.github.vlsi.mat.calcite.QueryCancellation; + import com.google.common.collect.FluentIterable; import com.google.common.collect.ImmutableList; import com.google.common.primitives.Ints; @@ -54,7 +56,7 @@ public Enumerator enumerator() { } }); - return Linq4j.iterableEnumerator(it); + return new CancellableEnumerator<>(Linq4j.iterableEnumerator(it)); } }; } @@ -72,4 +74,38 @@ public RelDataType getRowType(RelDataTypeFactory typeFactory) { Collections.singletonList("@ID") ); } + + /** + * Stops the scan when its query is cancelled ({@link QueryCancellation}), e.g. from MAT's progress view. Calcite + * checks its own cancel flag only between result rows, so without this an aggregation over a large class could not + * be stopped until it had read every object. + */ + static final class CancellableEnumerator implements Enumerator { + private final Enumerator delegate; + + CancellableEnumerator(Enumerator delegate) { + this.delegate = delegate; + } + + @Override + public T current() { + return delegate.current(); + } + + @Override + public boolean moveNext() { + QueryCancellation.check(); + return delegate.moveNext(); + } + + @Override + public void reset() { + delegate.reset(); + } + + @Override + public void close() { + delegate.close(); + } + } } diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AllTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AllTests.java index 0dec3b0..0e3785f 100644 --- a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AllTests.java +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/AllTests.java @@ -8,7 +8,11 @@ BasicQueriesTests.class, GetByKeyTests.class, TableFunctionsTests.class, - CollectionsFunctionsTests.class + CollectionsFunctionsTests.class, + RowSetTableTests.class, + SqlForObjectsTests.class, + DateFunctionsTests.class, + QueryCancellationTests.class }) public class AllTests { } diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java index bba2f97..e66a521 100644 --- a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/BasicQueriesTests.java @@ -1,10 +1,38 @@ package com.github.vlsi.mat.tests.calcite; +import com.github.vlsi.mat.calcite.Query; +import com.github.vlsi.mat.calcite.SqlIdentifiers; + +import org.eclipse.mat.util.VoidProgressListener; +import org.junit.Assert; import org.junit.Test; import java.sql.SQLException; +import java.util.concurrent.CancellationException; public class BasicQueriesTests extends SampleHeapDumpTests { + /** Cancel in MAT's progress view, or closing the SQL tab, stops a query that is scanning the heap. */ + @Test + public void aCancelledProgressListenerStopsTheQuery() throws Exception { + Query query = new Query(); + query.snapshot = getSnapshot(); + query.sql = "select count(*) from \"java.lang.String\""; + VoidProgressListener listener = new VoidProgressListener(); + listener.setCanceled(true); + try { + query.execute(listener); + Assert.fail("expected the cancelled query to stop"); + } catch (Exception e) { + Throwable t = e; + while (t != null && !(t instanceof CancellationException)) { + t = t.getCause(); + } + if (t == null) { + throw e; + } + } + } + @Test public void testRegularClassName() throws SQLException { @@ -51,6 +79,20 @@ public void testInstanceOfQuotedClassName() throws SQLException { execute("select * from \"instanceof.java.util.HashMap$Node\"", 1); } + @Test + public void testInstanceOfFullClassName() throws SQLException { + execute("select * from instanceof.\"java.util.HashMap$Node\"", 1); + } + + @Test + public void testInstanceOfReservedWordPackage() throws SQLException { + // REF is reserved in Calcite's parser, so the package part needs quotes; the full name form does not + execute("select count(*) from instanceof.java.lang.\"ref\".Reference", 1); + execute("select count(*) from instanceof.\"java.lang.ref.Reference\"", 1); + Assert.assertEquals("java.lang.\"ref\".Reference", SqlIdentifiers.quotePath("java.lang.ref.Reference")); + Assert.assertEquals("java.util.\"HashMap$Node\"", SqlIdentifiers.quotePath("java.util.HashMap$Node")); + } + @Test public void testStackTraceCount() throws SQLException { returnsInOrder("select count(*) CNT from native.ThreadStackFrames", diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/DateFunctionsTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/DateFunctionsTests.java new file mode 100644 index 0000000..17a7469 --- /dev/null +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/DateFunctionsTests.java @@ -0,0 +1,58 @@ +package com.github.vlsi.mat.tests.calcite; + +import org.junit.Test; + +import java.sql.SQLException; + +/** + * Date functions on the sample dump (a JDK 8 dump with java.util.Date and GregorianCalendar objects) and on plain + * numbers. The java.time and java.sql types were checked by hand against a dump made for it; see MIGRATION-NOTES. + */ +public class DateFunctionsTests extends SampleHeapDumpTests { + + @Test + public void utilDateInUtcAndInTheDumpedJvmZone() throws SQLException { + returnsInOrder("select toDateTime(this) d, toDateTime(this, 'jvm') jvm, toEpochMillis(this) ms " + + "from \"java.util.Date\" limit 1", + "d|jvm|ms", + "1934-11-08 21:09:09.328|1934-11-09 00:09:09.328|-1109127050672"); + } + + @Test + public void calendarInItsOwnZone() throws SQLException { + returnsInOrder("select toDateTime(this) d, toDateTime(this, 'UTC') utc, toEpochMillis(this) ms " + + "from \"java.util.GregorianCalendar\" limit 1", + "d|utc|ms", + "1934-11-09 00:09:09.328 Europe/Moscow|1934-11-08 21:09:09.328|-1109127050672"); + } + + @Test + public void epochNumbers() throws SQLException { + returnsInOrder("select millisToDateTime(0) a, millisToDateTime(0, 'Europe/Madrid') b, " + + "secondsToDateTime(1790853071) c, toDateTime(1790853071224) d", + "a|b|c|d", + "1970-01-01 00:00:00.000|1970-01-01 01:00:00.000|2026-10-01 11:11:11.000|2026-10-01 11:11:11.224"); + } + + @Test + public void durations() throws SQLException { + returnsInOrder("select toDuration(93784567) a, toDuration(-1500) b, toDuration(250) c", + "a|b|c", + "1d 2h 3m 4.567s|-1.500s|0.250s"); + } + + @Test + public void extremeDurationsDoNotOverflow() throws SQLException { + // Long.MAX_VALUE milliseconds is about 292 million years: seconds * 10^9 used to wrap around + returnsInOrder("select toDuration(9223372036854775807) a, toDuration(-9223372036854775808) b", + "a|b", + "106751991167d 7h 12m 55.807s|-106751991167d 7h 12m 55.808s"); + } + + @Test + public void notADate() throws SQLException { + returnsInOrder("select toDateTime(this) d, toEpochMillis(this) ms from java.util.HashMap limit 1", + "d|ms", + "null|null"); + } +} diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/QueryCancellationTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/QueryCancellationTests.java new file mode 100644 index 0000000..8bf29ac --- /dev/null +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/QueryCancellationTests.java @@ -0,0 +1,61 @@ +package com.github.vlsi.mat.tests.calcite; + +import com.github.vlsi.mat.calcite.QueryCancellation; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.concurrent.CancellationException; +import java.util.concurrent.atomic.AtomicBoolean; + +public class QueryCancellationTests { + @Test + public void checkThrowsOnlyInsideACancelledScope() { + QueryCancellation.check(); // no scope: nothing to cancel + AtomicBoolean cancelled = new AtomicBoolean(); + try (QueryCancellation.Scope scope = QueryCancellation.enter(cancelled::get)) { + QueryCancellation.check(); + cancelled.set(true); + try { + QueryCancellation.check(); + Assert.fail("expected a CancellationException"); + } catch (CancellationException expected) { + // the scope's flag is set + } + } + QueryCancellation.check(); // the scope is closed: its flag no longer applies + } + + @Test + public void nestedScopesRestoreTheOuterFlag() { + AtomicBoolean outer = new AtomicBoolean(true); + try (QueryCancellation.Scope a = QueryCancellation.enter(outer::get)) { + try (QueryCancellation.Scope b = QueryCancellation.enter(() -> false)) { + QueryCancellation.check(); + } + try { + QueryCancellation.check(); + Assert.fail("the outer flag should apply again"); + } catch (CancellationException expected) { + // restored + } + } + } + + @Test + public void aFlagBelongsToItsThread() throws Exception { + try (QueryCancellation.Scope scope = QueryCancellation.enter(() -> true)) { + Throwable[] failure = new Throwable[1]; + Thread other = new Thread(() -> { + try { + QueryCancellation.check(); + } catch (Throwable e) { + failure[0] = e; + } + }); + other.start(); + other.join(); + Assert.assertNull("another thread saw this thread's flag", failure[0]); + } + } +} diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/RowSetTableTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/RowSetTableTests.java new file mode 100644 index 0000000..eb8aa61 --- /dev/null +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/RowSetTableTests.java @@ -0,0 +1,32 @@ +package com.github.vlsi.mat.tests.calcite; + +import com.github.vlsi.mat.calcite.Query; +import com.github.vlsi.mat.calcite.RowSetTable; + +import org.eclipse.mat.util.VoidProgressListener; +import org.junit.Assert; +import org.junit.Test; + +public class RowSetTableTests extends SampleHeapDumpTests { + + private RowSetTable run(String sql) throws Exception { + Query query = new Query(); + query.snapshot = getSnapshot(); + query.sql = sql; + return (RowSetTable) query.execute(new VoidProgressListener()); + } + + @Test + public void objectRowsHaveIcons() throws Exception { + RowSetTable table = run("select s.this from java.lang.String s limit 1"); + Assert.assertEquals(1, table.getRowCount()); + Assert.assertNotNull("icon for a String instance row", table.getIcon(table.getRow(0))); + } + + @Test + public void scalarRowsHaveNoIcons() throws Exception { + RowSetTable table = run("select count(*) from java.lang.String"); + Assert.assertEquals(1, table.getRowCount()); + Assert.assertNull("icon for a row without objects", table.getIcon(table.getRow(0))); + } +} diff --git a/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/SqlForObjectsTests.java b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/SqlForObjectsTests.java new file mode 100644 index 0000000..8ee485e --- /dev/null +++ b/MatCalciteTest/src/com/github/vlsi/mat/tests/calcite/SqlForObjectsTests.java @@ -0,0 +1,93 @@ +package com.github.vlsi.mat.tests.calcite; + +import com.github.vlsi.mat.calcite.SqlForObjects; + +import org.eclipse.mat.query.IContextObject; +import org.eclipse.mat.query.IContextObjectSet; +import org.eclipse.mat.snapshot.model.IClass; +import org.junit.Assert; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +public class SqlForObjectsTests extends SampleHeapDumpTests { + + private IClass classOf(String name) throws Exception { + return getSnapshot().getClassesByName(name, false).iterator().next(); + } + + private static IContextObject object(int id) { + return () -> id; + } + + /** Number of result rows of the generated SQL (the CSV also has a header line). */ + private int rows(String sql) throws Exception { + return executeToCSV(sql).size() - 1; + } + + @Test + public void selectedObjects() throws Exception { + int[] ids = classOf("java.lang.String").getObjectIds(); + List elements = new ArrayList<>(); + for (int i = 0; i < 3; i++) { + elements.add(object(ids[i])); + } + String sql = SqlForObjects.build(getSnapshot(), elements); + Assert.assertTrue(sql, sql.contains("from \"java.lang.String\"")); + Assert.assertEquals(sql, 3, rows(sql)); + } + + private static IContextObjectSet classRow(IClass cls, int[] instances) { + return new IContextObjectSet() { + @Override + public int getObjectId() { + return cls.getObjectId(); + } + + @Override + public int[] getObjectIds() { + return instances; + } + + @Override + public String getOQL() { + return null; + } + }; + } + + /** A class row of a partial histogram, e.g. of a retained set, stands for some of the class's instances only. */ + @Test + public void partialClassRowSelectsItsObjectsOnly() throws Exception { + IClass stringClass = classOf("java.lang.String"); + int[] some = Arrays.copyOf(stringClass.getObjectIds(), 5); + String sql = SqlForObjects.build(getSnapshot(), Collections.singletonList(classRow(stringClass, some))); + Assert.assertTrue(sql, sql.contains("getAddress(this) in (")); + Assert.assertEquals(sql, 5, rows(sql)); + } + + @Test + public void classRowSelectsAllInstances() throws Exception { + IClass threadClass = classOf("java.lang.Thread"); + int[] instances = threadClass.getObjectIds(); + String sql = SqlForObjects.build(getSnapshot(), Collections.singletonList(classRow(threadClass, instances))); + Assert.assertEquals("select *\n from \"java.lang.Thread\"\n", sql); + Assert.assertEquals(sql, threadClass.getNumberOfObjects(), rows(sql)); + } + + @Test + public void severalClassesGiveOneStatementEach() throws Exception { + int thread = classOf("java.lang.Thread").getObjectIds()[0]; + int string = classOf("java.lang.String").getObjectIds()[0]; + String sql = SqlForObjects.build(getSnapshot(), Arrays.asList(object(thread), object(string))); + Assert.assertTrue(sql, sql.startsWith("-- Objects of several classes")); + String[] statements = sql.substring(sql.indexOf("\n\n") + 2).split("\n(?=select)"); + Assert.assertEquals(sql, 2, statements.length); + for (String statement : statements) { + Assert.assertEquals(statement, 1, rows(statement)); + } + } +}