0.0.46 release

v0.0.46 Release Notes

Bugs

• Pull in version bounds for third-party packages from the master branch. This addresses an immediate issue where Synapse fails to work with a pypi installation and Tornado 5.0.0 is installed.

0.0.45 release

Synapse v0.0.45 Release Notes

New Features

• #661 - Added the inet:group form and type. This is analogous to the inet:user form for tracking a given group name.

Enhancements

• #666 - Add ps:person:name:en and various name:en subprops props to distinguish between localized and English versions of names.
• #661 - The inet:web:group form was made more robust. The second field of the sepr type was renamed to :id to represent a site specific identifier, freeing the :name property up to represent a descriptive name of the group. :name:en was added to provide for a English version of the name. :signup, :signup:ipv4 and signup:ipv6 props were added to track when groups were created. :loc and :latlong props were added to allow for some location based tracking. :seen:min and :seen:max secondary properties were added to track when a group was active.
• #662 - The python msgpack library was version locked to 0.5.1.
• #663 - Added :name:en and :realname:en secondary props to inet:web:acct for tracking English specific versions of a name.

Bugs

• #624 - The ps:person was missing the :name:middle field. This has been added.
• #660 - The ou:member form was broken. This has been changed to be a Comp type and had the secondary property :end added to it.

0.0.44 release

Synapse v0.0.44 Release Notes

Notices related to v0.0.44

The minimum version of msgpack used by Synapse has been upgraded to be at least 0.5.0. This is complicated by the fact that the msgpack-python package was renamed to msgpack by its maintainers. Installation of Synapse from PyPi using the sdist and wheel packages should upgrade msgpack in a working state, and docker image users should not be affected by this change. Users which directly use a checkout of the Synapse Github repository should uninstall msgpack-python and then install msgpack. This can be done with the following commands (you may need to adjust them for your installation; accounting for aliases or the use of python3, etc):

• python -m pip uninstall msgpack-python
• python -m pip install msgpack

The synapse.lib.heap.Heapfile and synapse.axon.Axon structures are planned to undergo a significant rewrite, possibly landing as early as v0.0.45. These rewrites will not be backwards compatible, in order to account for design issues in these structures. If you are using these classes for doing data storage, please reach out to us via Slack as soon as possible (see https://github.com/vertexproject/synapse/blob/master/README.rst for a link to join our Synapse Slack chatroom).

New Features

• #592 - Added GUID type it:auth:passwdhash to store password hashes without requiring the plaintext password.
• #604 - Added synapse.lib.atomic.Counter() class as a thread safe counter object.
• #595 - Added the rcode secondary property to the inet:dns:look form to allow encoding the DNS response code for a given lookup.
• #595 - The it:dev:regkey type is now a lowercased string. Existing it:dev:regkey and it:dev:regval:key values will be automatically lowercased as a model migration.
• #607 - Added Axon.getAxonStatus() and AxonHost.getAxonHostStatus() APIs to get runtime information about Axons.

Enhancements

• #591 - Removed the Cortex CLI command nextseq in favor of using the Storm macro command nexttag().
• #591 - Add Storm test coverage.
• #593 - Added puts() API to the synapse.lib.fifo.Fifo object to the Fifo structure to allow for bulk object adding.
• #596 - Added Axon tests to show a clone sync operation restarting after a persist offset file is removed.
• #601 - Ensure the synapse.lib.heap.Heap file storage only grows upward and cannot be resized down by replaying a resize event.
• #602 - Set backoff / backoff-retry values for the Drone ci configuration for the git plugin.
• #603 - The synapse.lib.tags.tufoHasTag() and synapse.lib.tufo.tagged() APIs were equivalent functions. tufoHasTag has been removed.
• #598, #609 - Updated msgpack-python>=0.4.8 to msgpack=>0.5.1 as a dependency.
• #598 - Updated synaspe.lib.msgpack.Unpk msgpack unpacker helper to use the new tell() API introduced in msgpack 0.5.0.
• #598 - Added a future-proofing msgpack test to ensure we break when the msgpack python API eventually changes to assuming a strict utf8 compliance and can then versionlock our use of msgpack.
• #605 - Added simple integrity checking to the synapse.lib.heap.HeapFile structure.

Bugs

• #591 - synapse.lib.cache.KeyCache behavior updated to no longer store cache valus if they are None. If None was cached, that value was unable to be updated later.
• #594, #597 - Fix Ingest user docs which were using outdated forms. Thanks @tn3rt for the bug report!
• #599 - Fix AxonMixin.eatfd() and AxonMixin.eatbytes() to allow file descriptors and bytes which have zero bytes to be consumed.

Documentation

• #605 - Rewrote API docstrings for synapse.lib.heap.HeapFile.

0.0.43 release

Synapse v0.0.43 Release Notes

New Features

• #590 - Added synapse.lib.remcycle.fetch() API which allows retrieving a URL using Tornado and executing a callback to process the response.

Enhancements

• #589 - Added the ability to set a tags directive in a Ingest forms definition; which allows setting per-form tags without adding the tags to the current or child scopes.

0.0.42 release

Synapse v0.0.42 Release Notes

New Features

• #588 - Added a RFC2822 address parser and type/form, inet:rfc2822:addr. This normalizes and parses string identifiers and attempts to extract email addresses as a secondary property.

Bugs

• #587 - Make synapse.lib.msgpack helper functions resilient to unmatched unicode surrogate characters. This also affects the synapse.lib.socket.Socket and synapse.lib.persist.Dir msgpack unpackers. This is done by passing unicode_errors='surrogatepass' to the msgpack pack/unpack functions. This makes synapse more resilient to malformed string data which may be encountered in the real world.

Documentation

• #587 - Removed synapse.statemach. It was previously used for providing object persistence at an API level but has been unused within Synapse and was generally not a safe tool to use for third party use.

0.0.41 release

Synapse v0.0.41 Release Notes

New Features

• #574 - Added EventBus.onWith() API. This is a context manager which acts like .on(), but the callback is removed when the context manager is exited.
• #575 - Added synapse.lib.iq.CmdGenerator() class to allow testing CLI command loops using unittest mock.
• #577 - Added synapse.lib.certdir.genClientCert() API to allow creation of a PKCS12 certificate bundle for a user certificate, private key and CA cert.
• #577 - Added a --p12 option to the easycert tool. This allows a user to bundle their certificate, private key and CA cert into a PKC12 formatted file.
• #578 - Added the Storm macro syntax <- to represent a join() operation.

Enhancements

• #576 - The Daemon now fini's EventBus() objects in LIFO order when it is fini'd. In other words, objects created in a dmon configuration are torn down in reverse order that they are created.
• #576, #581 - Increased test coverage for CLI related tests.
• #577 - Updated tests for synapse.lib.certdir to ensure that the certificates made by certdir were correct.
• #577 - Updated tests for synapse.tools.easycert to ensure that the certificates made by easycert were correct.
• #578 - Updated the storm join() operator syntax to behave exactly like the pivot() operator with respect to source and destination properties. join() still is an additive operator which does not consume the source nodes.
• #583 - synapse.lib.remcycle.Hypnos now registers and persists ingest definitions as syn:ingest nodes in its Cortex.

Bugs

• #579 - Fix a reference to onCtx (the original name for the EventBus.onWith() function).
• #584 - Cortex Storage backings had different behaviors when limit=0 was passed to functions which joined rows together to make tufos. This has been fixed, so that a limit=0 API parameter will return 0 rows.

Documentation

• #573 - Added telepath docstrings for evalurl().
• #577 - Rewrote API documentation for synapse.lib.certdir.
• #586 - Updated the Storm join() documentation to reflect changes to its arguments.

0.0.40 release

Synapse v0.0.40 Release Notes

New Features

• #564 - Added a int:2:str10 typecast to convert a integer value to the base10 string value.
• #569 - The Telepath Proxy object now fires the local event tele:sock:runsockfini when the Proxy is attempting to reconnect to its remote Daemon. This can be hooked by implementers to detect the loss of an existing connection to a remote Daemon.
• #572 - The synapse.axon.AxonCluster helper object is now a subclass of EventBus. This will now automatically close its corresponding SvcProx instance on its own .fini().
• #572 - Added the synapse.lib.iq.objhierarchy() function to extract a type hierarchy from nested tufo-style objects. This can be a useful for debugging.

Enhancements

• #565 - Increase Tornado related test timeouts from 5 to 30 seconds.
• #570 - Additional debug logging added to synapse.axon.Axon actions.
• #572 - Additional debug logging added to synapse.lib.persist.Dir actions.

Bugs

• #561 - When a telepath Proxy object is made, by default it will create its own socket Plex object. This Plex is now fini’d when the proxy is fini’d.
• #568 - Add synapse.glob to synapse.lib.socket imports to fix a missing reference.
• #567- The synapse.lib.persist.Dir.items() method yielded offsets which could switch from being relative offsets to being absolute offsets. This has been changed to always yield absolute offsets.
• #570 - The synapse.axon.Axon._fireAxonClone thread now uses the tele:sock:runsockfini event to detect that a disconnect has occurred for its remote Proxy, and uses that to break out of its innermost synapse.lib.persist.Dir.items() loop. The previous behavior would fail to reconnect if part of an Axon cluster went down and then came back up, as the Proxy contained potentially invalid link information instead of getting an updated link tufo from the Axon’s ServiceBus.
• #571 - Fini Axon threads after calling self.syncdir.fini() to ensure that any threads which are in the real time event pump from the syncdir can safely close their resources before attempting to stop any threads.

Documentation

• #572 - Miscellaneous API docstring improvements.

0.0.39 release

Synapse v0.0.39 Release Notes

New Features

• #553 - Synapse Docker images will now have -debug images available as well for all master and tagged builds. This image is built off of the vertexproject/pydebug:3.6-slim image. This image has full GDB support for Python enabled. See https://github.com/vertexproject/pydebug for more information.

Enhancements

• #551 - Make 4096 bit RSA keys by default when using the synaspse.tools.easycert tool. Thanks @galyac!
• #550 - Add additional debug logging to Synapse when starting up a Cortex, executing remote methods via Telepath and around SSL errors.
• #554 - Validate the Storm tree() recurlim argument is greater than or equal to zero.
• #557 - inet:dns:look:tcp4:ipv4 and inet:dns:look:tcp4:udp4 sub properties of port and ipv4 have been added, so a DNS response can be recorded from a server even if the server source port is unknown.
• #559 - Add a limit parameter to the Storm totags() operator.
• #560 - Add unit test for synapse.lib.dark.

Bugs

• #555 - Set the x509 extension number of certificates made by synaspse.tools.easycert to version 3 (0x02). This allows Chrome to validate the certificates starting in Chrome 63.
• #558 - When an exception is raised during the synapse.lib.socket.Plex._plexMainLoop call to select.select(), perform a check to see if any sockets are fini'd or have fileno() == -1 and perform a cleanup operation on that socket. This can reduce a race condition where a bad socket can get wedged and prevent the multiplexor from working correctly.
• #558 - Change the synapse.link.ssl.Socket.send() implementation to call self.sock.send() itself instead of calling through the default Socket.send() method. This allows us to properly catch SSLWantReadError and SSLWantWriteError exceptions; instead of allowing teh default method to catch them as OSError exceptions and fini the socket. The previous behavior lead to SSL Sockets being unstable.
• #556 - Fix the Storm filter function for +range() so that filter operator works properly.
• #562 - Fix the Storm filter function for +in() so that filter operator works properly.

Documentation

• #552 - Add documentation for the Storm operators to the user guide: pivot(), join(), refs(), fromtags(), totags(), jointags(), and tree().

0.0.38 release

Synapse v0.0.38 Release Notes

New Features

• #545 - Added storm macro function get:tasks and an API for introspecting tasks which have been registered on a Cortex.

Enhancements

• #544 - Added new fields (url and whois:fqdn) to whois:contact.
• #547 - Enabled pyup monitoring for Synapse release notes and added config file to disable pyup update checks.
• #549 - Removed cryptography from setup.py.

Documentation

• #548 - Added CHANGELOG.md to maintain release notes within the repository.

0.0.37 release

Synapse v0.0.37 Release Notes

New Features

• #542 - The Daemon now automatically calls item.fini() for items which are made from a Daemon configuration which are EventBus objects when the Daemon is fini()'d. This allows the Daemon to tear down all instances of eventbus objects which it makes without having to share the object and set the onfini option when configuring the share.

Enhancements

• #541 - Added exe, proc and host secondary properties to the inet:dns:look format to allow capturing a DNS lookup which may have originated from a file, process or a host.
• #540 - When the socket multiplexer does an accept() call, the remote ip and port are logged at the debug (logging.DEBUG) log level.

Bugs

• #58, 537 - Fixed IPv6 type norm() operations for OSX by using the ipaddress library instead of the s_socket.inet_ntop() function. Thanks @MichaelSquires !
• #543 - Recent changes to pytest (included in the base image used for doing CI testing) changed how logging is performed. This disables those pytest changes.