fix: optimize mempool transaction reads and writes #1781

rafaelcr · 2023-12-21T22:57:45Z

Delete mempool_digest materialized view, keep data in chain_tip table
Change the digest algorithm from a hash to a simple "updated at" timestamp so mempool writes are extremely cheap
Keep tx_count column to avoid running ad-hoc counts over unpruned txs
Insert mempool txs in batches

github-actions · 2023-12-21T23:01:49Z

Vercel deployment URL: https://stacks-blockchain-2cvh5ttuk-blockstack.vercel.app 🚀

codecov · 2023-12-21T23:57:37Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (da4cd56) 71.70% compared to head (cd4afff) 71.67%.
Report is 7 commits behind head on nakamoto.

Additional details and impacted files

@@             Coverage Diff              @@
##           nakamoto    #1781      +/-   ##
============================================
- Coverage     71.70%   71.67%   -0.03%     
============================================
  Files            89       89              
  Lines         11706    11687      -19     
  Branches       2593     2589       -4     
============================================
- Hits           8394     8377      -17     
+ Misses         3162     3160       -2     
  Partials        150      150

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/datastore/pg-write-store.ts

src/tests/mempool-tests.ts

zone117x

Left a comment about adding another test, otherwise LGTM!

* fix: change mempool_digest into a table * fix: change digest to be last updated timestamp * fix: build * fix: update count on reconcile * test: mempool renconcile

## [7.4.0-nakamoto.11](v7.4.0-nakamoto.10...v7.4.0-nakamoto.11) (2024-01-03) ### Features * create `/extended/v2/burn-blocks/:height_or_hash/blocks` endpoint ([#1782](#1782)) ([20466a1](20466a1)) ### Bug Fixes * optimize mempool transaction reads and writes ([#1781](#1781)) ([3a02f57](3a02f57))

blockstack-devops · 2024-01-03T15:26:38Z

🎉 This PR is included in version 7.4.0-nakamoto.11 🎉

The release is available on:

Your semantic-release bot 📦🚀

## [7.4.0-beta.1](v7.3.7-beta.1...v7.4.0-beta.1) (2024-01-03) ### Features * add `/extended/v2/mempool/fees` endpoint ([#1795](#1795)) ([ea9c378](ea9c378)) ### Bug Fixes * optimize mempool transaction reads and writes ([#1781](#1781)) ([#1792](#1792)) ([2700642](2700642))

## [7.4.0](v7.3.6...v7.4.0) (2024-01-04) ### Features * add `/extended/v2/mempool/fees` endpoint ([#1795](#1795)) ([ea9c378](ea9c378)) ### Bug Fixes * convert `chain_tip` materialized view into a table ([#1789](#1789)) ([0211932](0211932)), closes [#1751](#1751) * optimize mempool transaction reads and writes ([#1781](#1781)) ([#1792](#1792)) ([2700642](2700642)) * release pino logger and mempool nonces ([16d3593](16d3593)) * update client code, fix mempool fee return type ([#1797](#1797)) ([9853e29](9853e29))

## [7.6.0-nakamoto.1](v7.5.0...v7.6.0-nakamoto.1) (2024-01-09) ### Features * `GET /extended/v1/burn_block` ([#1766](#1766)) ([cb38b68](cb38b68)) * add `/extended/v2/blocks/:height_or_hash` ([#1774](#1774)) ([e532a5e](e532a5e)) * add `/extended/v2/blocks` endpoint with burn block filters ([#1769](#1769)) ([ceb7be0](ceb7be0)) * add `tx_count` property to `/extended/v2/blocks` ([#1778](#1778)) ([da4cd56](da4cd56)) * add dataset store ([4211328](4211328)) * add step to compile duckdb for Alpine image ([0f40e14](0f40e14)) * better handling of raw events insertion ([bb70ca9](bb70ca9)) * create `/extended/v2/burn-blocks/:height_or_hash/blocks` endpoint ([#1782](#1782)) ([20466a1](20466a1)) * disable rosetta via an ENV var ([#1804](#1804)) ([2d2aee3](2d2aee3)) * event-replay new_block events handling ([1708b42](1708b42)) * event-replay new_burn_block events handling ([6c0f448](6c0f448)) * event-replay raw events handling ([81f43cf](81f43cf)) * event-replay remainder events handling ([3ede07f](3ede07f)) * event-replay supporting parallel insertions ([f33ecee](f33ecee)) * events folder as environment var ([701bd1a](701bd1a)) * ingestion for `TenureChange` and `NakamotoCoinbase` tx types ([#1753](#1753)) ([7c45f53](7c45f53)) * parallel processing using node cluster ([d02a7e8](d02a7e8)) * pox-4 support ([#1754](#1754)) ([285806f](285806f)) * processing raw events in parallel ([7a6f241](7a6f241)) * update to latest TenureChange tx payload ([#1767](#1767)) ([2afb65c](2afb65c)) ### Bug Fixes * add token offering ([8ef039e](8ef039e)) * allow contract-principals in `/extended/v1/address/:principal/mempool` endpoint [#1685](#1685) ([#1704](#1704)) ([163b76a](163b76a)) * better args handlling ([c77ac57](c77ac57)) * better path handling for workers ([1bd8f17](1bd8f17)) * changed processing order ([62a12bd](62a12bd)) * convert `chain_tip` materialized view into a table ([#1751](#1751)) ([04b71cc](04b71cc)) * do not load duckdb binary unless required ([#1776](#1776)) ([db859ae](db859ae)) * flaky test ([484d2ea](484d2ea)) * flaky test ([65175f5](65175f5)) * handle `Problematic` status in `/drop_mempool_tx` event ([#1790](#1790)) ([ce9b38f](ce9b38f)) * import statement in replay controller ([7a10cd8](7a10cd8)) * insert block transaction data in batches ([#1760](#1760)) ([bf99e90](bf99e90)) * lint ([01589ea](01589ea)) * lint ([82eadcb](82eadcb)) * lint ([8c67ae5](8c67ae5)) * move `/extended/v1/burn_block` to `/extended/v2/burn-blocks` ([#1772](#1772)) ([bf2ef0a](bf2ef0a)) * on attachments_new events processing ([0707313](0707313)) * optimize mempool transaction reads and writes ([#1781](#1781)) ([3a02f57](3a02f57)) * re-enable indexes when finishing the replay ([fc379eb](fc379eb)) * remove dangling promise ([62a48ae](62a48ae)) * remove deprecated token endpoints ([#1775](#1775)) ([18f74b7](18f74b7)) * revert configurable DB index type ([86154b2](86154b2)) * upgrade semver package to fix ReDoS vulnerability ([6b1605b](6b1605b)) * vercel preview builds ([#1783](#1783)) ([d36b1c2](d36b1c2))

blockstack-devops · 2024-01-09T14:39:08Z

🎉 This PR is included in version 7.6.0-nakamoto.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

## [7.6.0](v7.5.0...v7.6.0) (2024-01-09) ### Features * `GET /extended/v1/burn_block` ([#1766](#1766)) ([cb38b68](cb38b68)) * add `/extended/v2/blocks/:height_or_hash` ([#1774](#1774)) ([e532a5e](e532a5e)) * add `/extended/v2/blocks` endpoint with burn block filters ([#1769](#1769)) ([ceb7be0](ceb7be0)) * add `order_by` and `order` params to `/extended/v1/tx/mempool` ([#1810](#1810)) ([2d45b2e](2d45b2e)) * add `tx_count` property to `/extended/v2/blocks` ([#1778](#1778)) ([da4cd56](da4cd56)) * add dataset store ([4211328](4211328)) * add step to compile duckdb for Alpine image ([0f40e14](0f40e14)) * better handling of raw events insertion ([bb70ca9](bb70ca9)) * create `/extended/v2/burn-blocks/:height_or_hash/blocks` endpoint ([#1782](#1782)) ([20466a1](20466a1)) * disable rosetta via an ENV var ([#1804](#1804)) ([2d2aee3](2d2aee3)) * event-replay new_block events handling ([1708b42](1708b42)) * event-replay new_burn_block events handling ([6c0f448](6c0f448)) * event-replay raw events handling ([81f43cf](81f43cf)) * event-replay remainder events handling ([3ede07f](3ede07f)) * event-replay supporting parallel insertions ([f33ecee](f33ecee)) * events folder as environment var ([701bd1a](701bd1a)) * ingestion for `TenureChange` and `NakamotoCoinbase` tx types ([#1753](#1753)) ([7c45f53](7c45f53)) * parallel processing using node cluster ([d02a7e8](d02a7e8)) * pox-4 support ([#1754](#1754)) ([285806f](285806f)) * processing raw events in parallel ([7a6f241](7a6f241)) * support tenure_change in tx type filter queries ([#1808](#1808)) ([0831393](0831393)) * update to latest TenureChange tx payload ([#1767](#1767)) ([2afb65c](2afb65c)) ### Bug Fixes * add token offering ([8ef039e](8ef039e)) * allow contract-principals in `/extended/v1/address/:principal/mempool` endpoint [#1685](#1685) ([#1704](#1704)) ([163b76a](163b76a)) * better args handlling ([c77ac57](c77ac57)) * better path handling for workers ([1bd8f17](1bd8f17)) * changed processing order ([62a12bd](62a12bd)) * convert `chain_tip` materialized view into a table ([#1751](#1751)) ([04b71cc](04b71cc)) * do not load duckdb binary unless required ([#1776](#1776)) ([db859ae](db859ae)) * **docs:** URL query arrays should be formatted with `form` rather than comma-separated ([#1807](#1807)) ([e184fb5](e184fb5)) * flaky test ([484d2ea](484d2ea)) * flaky test ([65175f5](65175f5)) * handle `Problematic` status in `/drop_mempool_tx` event ([#1790](#1790)) ([ce9b38f](ce9b38f)) * import statement in replay controller ([7a10cd8](7a10cd8)) * insert block transaction data in batches ([#1760](#1760)) ([bf99e90](bf99e90)) * lint ([01589ea](01589ea)) * lint ([82eadcb](82eadcb)) * lint ([8c67ae5](8c67ae5)) * move `/extended/v1/burn_block` to `/extended/v2/burn-blocks` ([#1772](#1772)) ([bf2ef0a](bf2ef0a)) * on attachments_new events processing ([0707313](0707313)) * optimize mempool transaction reads and writes ([#1781](#1781)) ([3a02f57](3a02f57)) * re-enable indexes when finishing the replay ([fc379eb](fc379eb)) * remove dangling promise ([62a48ae](62a48ae)) * remove deprecated token endpoints ([#1775](#1775)) ([18f74b7](18f74b7)) * revert configurable DB index type ([86154b2](86154b2)) * support comma-separated strings in array query params ([#1809](#1809)) ([c9a4df8](c9a4df8)) * upgrade semver package to fix ReDoS vulnerability ([6b1605b](6b1605b)) * vercel preview builds ([#1783](#1783)) ([d36b1c2](d36b1c2))

blockstack-devops · 2024-01-09T17:23:58Z

🎉 This PR is included in version 7.6.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

fix: change mempool_digest into a table

736f1ee

rafaelcr requested a review from zone117x December 21, 2023 22:57

rafaelcr temporarily deployed to Preview December 21, 2023 22:58 — with GitHub Actions Inactive

fix: change digest to be last updated timestamp

f00aea0

rafaelcr temporarily deployed to Preview December 21, 2023 23:28 — with GitHub Actions Inactive

fix: build

833f378

rafaelcr temporarily deployed to Preview December 21, 2023 23:55 — with GitHub Actions Inactive

zone117x reviewed Jan 2, 2024

View reviewed changes

src/datastore/pg-write-store.ts Show resolved Hide resolved

fix: update count on reconcile

64c7998

rafaelcr temporarily deployed to Preview January 2, 2024 15:27 — with GitHub Actions Inactive

zone117x reviewed Jan 2, 2024

View reviewed changes

src/tests/mempool-tests.ts Show resolved Hide resolved

zone117x approved these changes Jan 2, 2024

View reviewed changes

test: mempool renconcile

cd4afff

rafaelcr temporarily deployed to Preview January 2, 2024 15:43 — with GitHub Actions Inactive

rafaelcr merged commit 3a02f57 into nakamoto Jan 2, 2024
30 checks passed

rafaelcr deleted the fix/mempool-digest branch January 2, 2024 15:59

blockstack-devops added the released on @nakamoto label Jan 3, 2024

blockstack-devops added the released label Jan 9, 2024

baby636 mentioned this pull request May 14, 2024

[Snyk] Fix for 3 vulnerabilities baby636/stacks-blockchain-api#93

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: optimize mempool transaction reads and writes #1781

fix: optimize mempool transaction reads and writes #1781

rafaelcr commented Dec 21, 2023 •

edited

Loading

github-actions bot commented Dec 21, 2023 •

edited

Loading

codecov bot commented Dec 21, 2023 •

edited

Loading

zone117x left a comment

blockstack-devops commented Jan 3, 2024

blockstack-devops commented Jan 9, 2024

blockstack-devops commented Jan 9, 2024

fix: optimize mempool transaction reads and writes #1781

fix: optimize mempool transaction reads and writes #1781

Conversation

rafaelcr commented Dec 21, 2023 • edited Loading

github-actions bot commented Dec 21, 2023 • edited Loading

codecov bot commented Dec 21, 2023 • edited Loading

Codecov Report

zone117x left a comment

Choose a reason for hiding this comment

blockstack-devops commented Jan 3, 2024

blockstack-devops commented Jan 9, 2024

blockstack-devops commented Jan 9, 2024

rafaelcr commented Dec 21, 2023 •

edited

Loading

github-actions bot commented Dec 21, 2023 •

edited

Loading

codecov bot commented Dec 21, 2023 •

edited

Loading