Fixes and verifies issue 42 (#44)

* Fixes and verifies issue 42

* Improving documentation.

AUTHORS

@@ -3,4 +3,4 @@ Daniel Lemire
 Kendall Willets
 Alexander Gallego
 @aqrit
-Vladimir Kazanov 
+Vladimir Kazanov

README.md

@@ -20,9 +20,9 @@ expects a Linux-like system. We have a CMake build.
 
 # Users
 
-This library is used by 
+This library is used by
 
- * [UpscaleDB](https://github.com/cruppstahl/upscaledb), 
+ * [UpscaleDB](https://github.com/cruppstahl/upscaledb),
  * Redis' [RediSearch](https://github.com/RedisLabsModules/RediSearch),
  * [Facebook Thrift](https://github.com/facebook/fbthrift),
  * [Trinity Information Retrieval framework](https://github.com/phaistos-networks/Trinity).
@@ -73,8 +73,10 @@ size_t compsize = streamvbyte_delta_encode(datain, N, compressedbuffer,0); // en
 streamvbyte_delta_decode(compressedbuffer, recovdata, N,0); // decoding (fast)
 ```
 You have to know how many integers were coded when you decompress. You can store this
-information along with the compressed stream.
+information along with the compressed stream. The
 
+During decoding, the library may read up to `STREAMVBYTE_PADDING` extra bytes
+from the input buffer (these bytes are read but never used).
 
 Signed integers
 -----------------

include/streamvbyte.h

@@ -10,6 +10,8 @@
 extern "C" {
 #endif
 
+#define STREAMVBYTE_PADDING 16
+
 // Encode an array of a given length read from in to bout in varint format.
 // Returns the number of bytes written.
 // The number of values being stored (length) is not encoded in the compressed stream,
@@ -28,17 +30,21 @@ size_t streamvbyte_encode_0124(const uint32_t *in, uint32_t length, uint8_t *out
 // return the maximum number of compressed bytes given length input integers
 // in the worst case we overestimate data bytes required by four, see below
 // for a function you can run upfront over your data to compute allocations
+// It includes the STREAMVBYTE_PADDING bytes.
 static inline size_t streamvbyte_max_compressedbytes(const uint32_t length) {
    // number of control bytes:
    size_t cb = (length + 3) / 4;
    // maximum number of control bytes:
    size_t db = (size_t) length * sizeof(uint32_t);
-   return cb + db;
+   return cb + db + STREAMVBYTE_PADDING;
 }
 
 // return the exact number of compressed bytes given length input integers
 // runtime in O(n) wrt. in; use streamvbyte_max_compressedbyte if you
 // care about speed more than potentially over-allocating memory
+// Our decoding functions may read (but not use) STREAMVBYTE_PADDING extra bytes beyond
+// the compressed data: the user needs to ensure that this region is allocated, and it
+// is not included by streamvbyte_compressedbytes.
 static inline size_t streamvbyte_compressedbytes(const uint32_t *in, uint32_t length) {
    // number of control bytes:
    size_t cb = (length + 3) / 4;
@@ -58,6 +64,9 @@ static inline size_t streamvbyte_compressedbytes(const uint32_t *in, uint32_t le
 // return the exact number of compressed bytes given length input integers
 // runtime in O(n) wrt. in; use streamvbyte_max_compressedbyte if you
 // care about speed more than potentially over-allocating memory
+// Our decoding functions may read (but not use) STREAMVBYTE_PADDING extra bytes beyond
+// the compressed data: the user needs to ensure that this region is allocated, and it
+// is not included by streamvbyte_compressedbytes.
 static inline size_t streamvbyte_compressedbytes_0124(const uint32_t *in, uint32_t length) {
    // number of control bytes:
    size_t cb = (length + 3) / 4;
@@ -76,7 +85,8 @@ static inline size_t streamvbyte_compressedbytes_0124(const uint32_t *in, uint32
 
 
 // Read "length" 32-bit integers in varint format from in, storing the result in out.
-// Returns the number of bytes read.
+// Returns the number of bytes read. We may read up to STREAMVBYTE_PADDING extra bytes
+// from the input buffer (these bytes are read but never used).
 // The caller is responsible for knowing how many integers ("length") are to be read:
 // this information ought to be stored somehow.
 // There is no alignment requirement on the "in" pointer.

include/streamvbytedelta.h

@@ -22,7 +22,8 @@ extern "C" {
 size_t streamvbyte_delta_encode(const uint32_t *in, uint32_t length, uint8_t *out, uint32_t prev);
 
 // Read "length" 32-bit integers in StreamVByte format from in, storing the result in out.
-// Returns the number of bytes read.
+// Returns the number of bytes read.  We may read up to STREAMVBYTE_PADDING extra bytes
+// from the input buffer (these bytes are read but never used). 
 // The caller is responsible for knowing how many integers ("length") are to be read: 
 // this information ought to be stored somehow.
 // There is no alignment requirement on the "in" pointer.