Repository navigation
feat: filtered accumulator enhancements observability and performance #25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -1395,6 +1395,16 @@ | |||||
| :when (join-node-matches node token fact bindings join-filter-fn (:env accum-condition))] | ||||||
| fact)) | ||||||
|
|
||||||
| (defn- stored-or-filtered | ||||||
| "Returns the facts in candidate-facts that pass the join filter for the given token. | ||||||
| A stored entry is always exact, so it is used when present. The node stores entries only for tokens it propagated. | ||||||
| Without an entry, which is also the case for sessions from older versions, the join filter runs." | ||||||
| [node memory token candidate-facts bindings join-filter-fn accum-condition] | ||||||
| (let [stored (mem/get-accum-filtered memory node (:matches token) bindings)] | ||||||
| (if (not= stored ::mem/no-accum-filtered) | ||||||
| stored | ||||||
| (vec (filter-accum-facts node token candidate-facts bindings join-filter-fn accum-condition))))) | ||||||
|
|
||||||
| ;; A specialization of the AccumulateNode that supports additional tests | ||||||
| ;; that have to occur on the beta side of the network. The key difference between this and the simple | ||||||
| ;; accumulate node is the join-filter-fn, which accepts a token and a fact and filters out facts that | ||||||
|
|
@@ -1434,6 +1444,7 @@ | |||||
|
|
||||||
| :when (some? converted-result)] | ||||||
|
|
||||||
| (mem/add-accum-filtered! memory node (:matches token) fact-bindings (vec filtered-facts)) | ||||||
| (send-accumulated node accum-condition accumulator result-binding token | ||||||
| converted-result fact-bindings transport memory listener)) | ||||||
|
|
||||||
|
|
@@ -1455,6 +1466,7 @@ | |||||
| (when (some? converted-result) | ||||||
| ;; Send the created accumulated item to the children. | ||||||
| (doseq [token tokens] | ||||||
| (mem/add-accum-filtered! memory node (:matches token) join-bindings []) | ||||||
| (send-accumulated node accum-condition accumulator result-binding token | ||||||
| converted-result join-bindings transport memory listener)))) | ||||||
|
|
||||||
|
|
@@ -1477,7 +1489,7 @@ | |||||
| (doseq [token tokens | ||||||
| [fact-bindings candidate-facts] grouped-candidate-facts | ||||||
|
|
||||||
| :let [filtered-facts (filter-accum-facts node token candidate-facts fact-bindings join-filter-fn accum-condition)] | ||||||
| :let [filtered-facts (stored-or-filtered node memory token candidate-facts fact-bindings join-filter-fn accum-condition)] | ||||||
|
|
||||||
| :when (or (seq filtered-facts) | ||||||
| ;; Even if there no filtered facts, if there are no new bindings an initial value | ||||||
|
|
@@ -1507,7 +1519,11 @@ | |||||
| (retract-accumulated node accum-condition accumulator result-binding token | ||||||
| converted-result join-bindings transport memory listener)))) | ||||||
|
|
||||||
| :else nil))) | ||||||
| :else nil) | ||||||
|
|
||||||
| ;; Removed after the retraction above, which reads these filtered facts. | ||||||
| (doseq [token tokens] | ||||||
| (mem/remove-accum-filtered! memory node (:matches token))))) | ||||||
|
|
||||||
| (get-join-keys [node] binding-keys) | ||||||
|
|
||||||
|
|
@@ -1552,7 +1568,7 @@ | |||||
| ;; since the final result is guaranteed to be the same. | ||||||
| :when (seq new-filtered-facts) | ||||||
|
|
||||||
| :let [previous-filtered-facts (filter-accum-facts node token previous-candidates bindings join-filter-fn accum-condition) | ||||||
| :let [previous-filtered-facts (stored-or-filtered node memory token previous-candidates bindings join-filter-fn accum-condition) | ||||||
|
|
||||||
| previous-accum-result-init (cond | ||||||
| (seq previous-filtered-facts) | ||||||
|
|
@@ -1591,6 +1607,12 @@ | |||||
| new-converted (when (some? accum-result) | ||||||
| (convert-return-fn accum-result))]] | ||||||
|
|
||||||
| ;; Store the facts even when the result does not change, e.g. acc/min with a new larger fact. | ||||||
| (if (some? new-converted) | ||||||
| (mem/add-accum-filtered! memory node (:matches token) bindings | ||||||
| (into (vec previous-filtered-facts) new-filtered-facts)) | ||||||
| (mem/remove-accum-filtered! memory node (:matches token) bindings)) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think I understand this case. There is no nil The idea is that this memory only stores filtered facts when the accumulator is able to be propagated? |
||||||
|
|
||||||
| (cond | ||||||
|
|
||||||
| ;; When both the new and previous result were nil do nothing. | ||||||
|
|
@@ -1648,9 +1670,11 @@ | |||||
| (doseq [;; Get all of the previously matched tokens so we can retract and re-send them. | ||||||
| token matched-tokens | ||||||
|
|
||||||
| :let [previous-facts (filter-accum-facts node token previous-candidates bindings join-filter-fn accum-condition) | ||||||
| :let [previous-facts (stored-or-filtered node memory token previous-candidates bindings join-filter-fn accum-condition) | ||||||
|
|
||||||
| new-facts (filter-accum-facts node token new-candidates bindings join-filter-fn accum-condition)] | ||||||
| ;; Equal facts give the same join filter result, so removing the retracted facts from the filtered facts | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is pretty nice. Looks like something that could have been done regardless of having the new storage here. Running the filter over again on the new candidates after right retract is not necessary. |
||||||
| ;; gives the same facts, in the same order, as filtering new-candidates. | ||||||
| new-facts (second (mem/remove-first-of-each facts previous-facts))] | ||||||
|
|
||||||
| ;; The previous matching elements are a superset of the matching elements after retraction. | ||||||
| ;; Therefore, if the counts before and after are equal nothing retracted actually matched | ||||||
|
|
@@ -1663,9 +1687,7 @@ | |||||
| ;; elements before and after a right-retraction cannot be different. | ||||||
| previous-result (do-accumulate accumulator previous-facts) | ||||||
|
|
||||||
| ;; TODO: Can we use the retract-fn here if present to improve performance? We'd also potentially | ||||||
| ;; avoid needing to filter facts twice above, since elements present both before and after retraction | ||||||
| ;; will given to the join-filter-fn twice (once when creating previous-facts and once when creating new-facts). | ||||||
| ;; TODO: Can we use the retract-fn here if present to improve performance? | ||||||
| ;; Note that any future optimizations here must ensure that the result propagated here is equal to the result | ||||||
| ;; that will be recreated as the previous result in right activate, and that this can be dependent on the order | ||||||
| ;; of candidates in the memory, since, for example (acc/all) can return both [A B] and [B A] but these are not equal. | ||||||
|
|
@@ -1687,6 +1709,10 @@ | |||||
| new-converted (when (some? new-result) | ||||||
| (convert-return-fn new-result))]] | ||||||
|
|
||||||
| (if (some? new-converted) | ||||||
| (mem/add-accum-filtered! memory node (:matches token) bindings (vec new-facts)) | ||||||
| (mem/remove-accum-filtered! memory node (:matches token) bindings)) | ||||||
|
|
||||||
| (cond | ||||||
|
|
||||||
| ;; When both the previous and new results are nil do nothing. | ||||||
|
|
@@ -1722,21 +1748,39 @@ | |||||
| ;; the join filter function. Since the :matches are not used in the join filter function and the bindings in the | ||||||
| ;; token will contain all bindings used in the "ancestor token" to join with these same facts, we can just pass the token | ||||||
| ;; as-is to the join filter. | ||||||
| ;; | ||||||
| ;; A read-only session has no compiled join-filter-fn, so it cannot filter the facts again. | ||||||
| ;; The activation methods above keep the filtered facts current with mem/add-accum-filtered!, and read them through stored-or-filtered. | ||||||
| ;; This lookup uses them first. The key is the :matches of the token that entered this node. | ||||||
| ;; That token's :matches is the prefix before this node's entry in the descendant token. | ||||||
| ;; Sessions serialized before the filtered facts were stored have none, so the join filter runs again. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How can the join filter run again if there is no join filter compiled to run? This comment contradicts itself for read-only sessions. |
||||||
| IAccumInspect | ||||||
| (token->matching-elements [this memory token] | ||||||
| (let [join-bindings (-> token :bindings (select-keys (get-join-keys this))) | ||||||
| fact-bindings (-> token :bindings (select-keys new-bindings)) | ||||||
| accum-reduced (mem/get-accum-reduced memory this join-bindings (merge join-bindings fact-bindings))] | ||||||
| accum-bindings (merge join-bindings fact-bindings) | ||||||
| accum-reduced (mem/get-accum-reduced memory this join-bindings accum-bindings) | ||||||
| matches (:matches token) | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| node-match-idx (first (keep-indexed (fn [i [_ node-id]] (when (= id node-id) i)) matches)) | ||||||
| filtered (if node-match-idx | ||||||
| (mem/get-accum-filtered memory this (subvec matches 0 node-match-idx) accum-bindings) | ||||||
| ::mem/no-accum-filtered)] | ||||||
| ;; The functionality to throw conditions with meaningful information assumes that all bindings in the token | ||||||
| ;; are meaningful to the join, which is not the case here since the token passed is from a descendant of this node, not | ||||||
| ;; this node. The generated error message also wouldn't make much sense in the context of session inspection. | ||||||
| ;; We could create specialized error handling here, but in reality most cases that cause errors here would also cause | ||||||
| ;; errors at rule firing time so the benefit would be limited. Nevertheless there would be some benefit and it is | ||||||
| ;; possible that we will do it in the future.. | ||||||
| (if (not= accum-reduced ::mem/no-accum-reduced) | ||||||
| (cond | ||||||
| (not= filtered ::mem/no-accum-filtered) | ||||||
| filtered | ||||||
|
|
||||||
| (not= accum-reduced ::mem/no-accum-reduced) | ||||||
| (for [fact accum-reduced | ||||||
| :when (join-filter-fn token fact fact-bindings {})] | ||||||
| fact) | ||||||
|
|
||||||
| :else | ||||||
| [])))) | ||||||
|
|
||||||
| ;; This lives here as it is both close to the node that it represents, and is accessible to both clj and cljs | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -48,6 +48,12 @@ | |
| ;; accumulated items on this node. | ||
| (get-accum-reduced-complete [memory node]) | ||
|
|
||
| ;; Returns the facts that passed the join filter of an accumulator node with a join filter. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just an observation: |
||
| ;; left-matches are the :matches of the token that entered the node. | ||
| ;; accum-bindings are the join bindings merged with the fact bindings. | ||
| ;; Returns ::no-accum-filtered when no facts are stored. The node and inspection read these facts. | ||
| (get-accum-filtered [memory node left-matches accum-bindings]) | ||
|
|
||
| ;; Returns insertions that occurred at the given node for the given token. | ||
| ;; Returns a sequence of the form | ||
| ;; [facts-inserted-for-one-rule-activation facts-inserted-for-another-rule-activation] | ||
|
|
@@ -80,6 +86,14 @@ | |
| ;; Removes the result of a reduced accumulator execution to the given memory and node. | ||
| (remove-accum-reduced! [memory node join-bindings fact-bindings]) | ||
|
|
||
| ;; Stores the facts that passed the join filter of an accumulator node with a join filter. | ||
| ;; See get-accum-filtered. | ||
| (add-accum-filtered! [memory node left-matches accum-bindings facts]) | ||
|
|
||
| ;; Removes the filtered facts stored for the token with the given left-matches at the given node. | ||
| ;; Without accum-bindings, removes them for all accum-bindings of that token. | ||
| (remove-accum-filtered! [memory node left-matches] [memory node left-matches accum-bindings]) | ||
|
|
||
| ;; Add a record that a given fact twas inserted at a given node with | ||
| ;; the given support. Used for truth maintenance. | ||
| ;; This should be called at most once per rule activation. | ||
|
|
@@ -457,7 +471,8 @@ | |
| ^Map beta-memory | ||
| ^Map accum-memory | ||
| ^Map production-memory | ||
| ^NavigableMap activation-map] | ||
| ^NavigableMap activation-map | ||
| ^Map accum-filtered-memory] | ||
|
|
||
| IMemoryReader | ||
| (get-rulebase [memory] rulebase) | ||
|
|
@@ -502,6 +517,9 @@ | |
| :fact-bindings fact-binding | ||
| :result reduced})) | ||
|
|
||
| (get-accum-filtered [memory node left-matches accum-bindings] | ||
| (get-in accum-filtered-memory [(:id node) left-matches accum-bindings] ::no-accum-filtered)) | ||
|
|
||
| (get-insertions [memory node token] | ||
| (get | ||
| (get production-memory (:id node) {}) | ||
|
|
@@ -629,6 +647,39 @@ | |
| (not-empty (dissoc fact-binding-accum-map fact-bindings)))) | ||
| (not-empty join-binding-accum-map))))) | ||
|
|
||
| (add-accum-filtered! [memory node left-matches accum-bindings facts] | ||
| (hm/compute! accum-filtered-memory (:id node) | ||
| (fn add-lmfm | ||
| [_ left-matches-filtered-map] | ||
| (if left-matches-filtered-map | ||
| (let [left-matches-filtered-map (->mutable-map left-matches-filtered-map)] | ||
| (hm/compute! left-matches-filtered-map left-matches | ||
| (fn add-abfm | ||
| [_ accum-bindings-filtered-map] | ||
| (assoc accum-bindings-filtered-map accum-bindings facts))) | ||
| left-matches-filtered-map) | ||
| {left-matches {accum-bindings facts}})))) | ||
|
|
||
| (remove-accum-filtered! [memory node left-matches] | ||
| (hm/compute-if-present! accum-filtered-memory (:id node) | ||
| (fn remove-lmfm | ||
| [_ left-matches-filtered-map] | ||
| (let [left-matches-filtered-map (->mutable-map left-matches-filtered-map)] | ||
| (hm/compute-if-present! left-matches-filtered-map left-matches | ||
| (fn remove-abfm [_ _] nil)) | ||
| (not-empty left-matches-filtered-map))))) | ||
|
|
||
| (remove-accum-filtered! [memory node left-matches accum-bindings] | ||
| (hm/compute-if-present! accum-filtered-memory (:id node) | ||
| (fn remove-lmfm | ||
| [_ left-matches-filtered-map] | ||
| (let [left-matches-filtered-map (->mutable-map left-matches-filtered-map)] | ||
| (hm/compute-if-present! left-matches-filtered-map left-matches | ||
| (fn remove-abfm | ||
| [_ accum-bindings-filtered-map] | ||
| (not-empty (dissoc accum-bindings-filtered-map accum-bindings)))) | ||
| (not-empty left-matches-filtered-map))))) | ||
|
|
||
| ;; The value under each token in the map should be a sequence | ||
| ;; of sequences of facts, with each inner sequence coming from a single | ||
| ;; rule activation. | ||
|
|
@@ -803,16 +854,21 @@ | |
| (persistent-maps persistent-vals production-memory) | ||
| (into {} | ||
| (map (juxt key (comp ->persistent-coll val))) | ||
| activation-map))))) | ||
| activation-map) | ||
| (persistent-maps persistent-vals accum-filtered-memory))))) | ||
|
|
||
| ;;; accum-filtered-memory holds the facts that passed the join filter of each accumulator node with a join filter. | ||
| ;;; The node and inspection read it. The node replaces these facts each time they change. | ||
| ;;; It is nil in memory restored from sessions serialized before this field existed. | ||
| (defrecord PersistentLocalMemory [rulebase | ||
| activation-group-sort-fn | ||
| activation-group-fn | ||
| alpha-memory | ||
| beta-memory | ||
| accum-memory | ||
| production-memory | ||
| activation-map] | ||
| activation-map | ||
| accum-filtered-memory] | ||
| IMemoryReader | ||
| (get-rulebase [memory] rulebase) | ||
|
|
||
|
|
@@ -858,6 +914,9 @@ | |
| :fact-bindings fact-binding | ||
| :result reduced})) | ||
|
|
||
| (get-accum-filtered [memory node left-matches accum-bindings] | ||
| (get-in accum-filtered-memory [(:id node) left-matches accum-bindings] ::no-accum-filtered)) | ||
|
|
||
| (get-insertions [memory node token] | ||
| (get | ||
| (get production-memory (:id node) {}) | ||
|
|
@@ -888,7 +947,8 @@ | |
| (.put treemap | ||
| activation-group | ||
| (->activation-priority-queue activations false))) | ||
| treemap)))) | ||
| treemap) | ||
| (->mutable-map (or accum-filtered-memory {}))))) | ||
|
|
||
| (defn local-memory | ||
| "Creates an persistent local memory for the given rule base." | ||
|
|
@@ -901,4 +961,5 @@ | |
| (hf/hash-map) | ||
| (hf/hash-map) | ||
| (hf/hash-map) | ||
| (hf/hash-map) | ||
| (hf/hash-map))) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
filter-accum-factsis one of the rare'ish places we use a lazy seq return. This now converts it to eager evaluation. Is there some rationale for this? I'd think thatfilter-accum-factscould useplatform/eager-forinstead offor, if we are just going to wrap it invechere now instead.It is actually a bit inconsistent in that we still use the lazy variant in the same AccumulateWithJoinFilterNode impl some of the time.