Skip to content

Commit c0f9fd4

Browse files
committed
Implement from_reader io::Read support
Add from_reader and use Read internally to improve the codegen of deku, the performance of derived parsers. Internally, this is implemented using the new src/reader.rs interface in deku parses. This has a waterfall effect on DekuReader implementations and the deku-derive/deku_read Derived impl. Previous usage with from_bytes and other API's are unchanged. There is somewhat of a performance hit for bit-only parses, but I find the major improvments in the bytes-wise parsing to be great enough to warrent this change. The following are some sample benchmarks: > critcmp before after group after before ----- ----- ------ deku_read_bits 1.24 845.8±14.29ns ? ?/sec 1.00 679.5±11.83ns ? ?/sec deku_read_byte 1.00 17.8±0.25ns ? ?/sec 2.12 37.8±3.35ns ? ?/sec deku_read_enum 1.00 15.3±0.15ns ? ?/sec 2.04 31.2±0.81ns ? ?/sec deku_read_vec 1.00 676.8±7.04ns ? ?/sec 2.16 1459.5±40.22ns ? ?/sec deku_write_bits 1.00 125.3±3.10ns ? ?/sec 1.04 130.2±11.12ns ? ?/sec deku_write_byte 1.00 161.6±4.86ns ? ?/sec 1.02 165.0±5.91ns ? ?/sec deku_write_enum 1.00 105.6±1.06ns ? ?/sec 1.03 109.0±7.20ns ? ?/sec deku_write_vec 1.00 4.6±0.04µs ? ?/sec 1.06 4.9±0.07µs ? ?/sec The above change removes DekuRead, and replaces it with DekuReader. This contains the from_reader_with_ctx. DekuContainerRead contains from_reader. The crate no_std_io was picked to supply a Read impl for the no_std feature. These are "re-export"ed. Add "`Read` enabled" docs to lib.rs Add tests/test_tuple.rs tests Update CHANGELOG.md to reflect changes and help migration to this usage Use llvm-cov in ci for the generation of more accurate coverage reports Update benchmarks to test more complex parser speeds Disable Miri CI Update ensure_no_std to work with new Read usage. Remove wee-alloc in favour of an updated crate for the allocator. Add inline to small functions
1 parent d1827d1 commit c0f9fd4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+2732
-1792
lines changed

.github/workflows/coverage.yml

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: Coverage
2+
3+
on: [pull_request, push]
4+
5+
jobs:
6+
coverage:
7+
runs-on: ubuntu-latest
8+
env:
9+
CARGO_TERM_COLOR: always
10+
steps:
11+
- uses: actions/checkout@v3
12+
- name: Install Rust
13+
run: rustup update stable
14+
- name: Install cargo-llvm-cov
15+
uses: taiki-e/install-action@cargo-llvm-cov
16+
- name: Generate code coverage
17+
run: cargo llvm-cov --workspace --codecov --output-path codecov.json
18+
- name: Upload coverage to Codecov
19+
uses: codecov/codecov-action@v3
20+
with:
21+
files: codecov.json
22+
fail_ci_if_error: true

.github/workflows/main.yml

+27-42
Original file line numberDiff line numberDiff line change
@@ -33,30 +33,31 @@ jobs:
3333
command: test
3434
args: --all
3535

36-
test_miri:
37-
name: Miri Test
38-
runs-on: ubuntu-latest
39-
steps:
40-
- uses: actions/checkout@v3
41-
- uses: actions-rs/toolchain@v1
42-
with:
43-
toolchain: nightly
44-
override: true
45-
components: miri
46-
- run: cargo miri test
47-
48-
test_miri_big_endian:
49-
name: Miri Test Big Endian
50-
runs-on: ubuntu-latest
51-
steps:
52-
- uses: actions/checkout@v3
53-
- uses: actions-rs/toolchain@v1
54-
with:
55-
toolchain: nightly
56-
override: true
57-
components: miri
58-
target: mips64-unknown-linux-gnuabi64
59-
- run: cargo miri test --target mips64-unknown-linux-gnuabi64
36+
# TODO: Enable Miri
37+
# test_miri:
38+
# name: Miri Test
39+
# runs-on: ubuntu-latest
40+
# steps:
41+
# - uses: actions/checkout@v3
42+
# - uses: actions-rs/toolchain@v1
43+
# with:
44+
# toolchain: nightly
45+
# override: true
46+
# components: miri
47+
# - run: cargo miri test
48+
#
49+
# test_miri_big_endian:
50+
# name: Miri Test Big Endian
51+
# runs-on: ubuntu-latest
52+
# steps:
53+
# - uses: actions/checkout@v3
54+
# - uses: actions-rs/toolchain@v1
55+
# with:
56+
# toolchain: nightly
57+
# override: true
58+
# components: miri
59+
# target: armebv7r-none-eabi
60+
# - run: cargo miri test --target armebv7r-none-eabi
6061

6162
examples:
6263
name: Examples
@@ -111,7 +112,8 @@ jobs:
111112
with:
112113
toolchain: nightly
113114
override: true
114-
- run: cd ensure_no_std && cargo run --release
115+
target: thumbv7em-none-eabihf
116+
- run: cd ensure_no_std && cargo build --release --target thumbv7em-none-eabihf
115117

116118
ensure_wasm:
117119
name: Ensure wasm
@@ -126,20 +128,3 @@ jobs:
126128
with:
127129
version: 'latest'
128130
- run: cd ensure_wasm && wasm-pack build --target web && wasm-pack test --node
129-
130-
coverage:
131-
name: Coverage
132-
runs-on: ubuntu-latest
133-
container:
134-
image: xd009642/tarpaulin:develop
135-
options: --security-opt seccomp=unconfined
136-
steps:
137-
- name: Checkout repository
138-
uses: actions/checkout@v3
139-
140-
- name: Generate code coverage
141-
run: |
142-
cargo tarpaulin --verbose --all-features --workspace --timeout 120 --out Xml
143-
144-
- name: Upload to codecov.io
145-
uses: codecov/codecov-action@v1

CHANGELOG.md

+129
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,135 @@
22

33
## [Unreleased]
44

5+
## Changes
6+
[#352](https://github.com/sharksforarms/deku/pull/352) added a new function `from_reader` that uses `io::Read`.
7+
`io::Read` is also now used internally, bringing massive performance and usability improvements.
8+
9+
### New `from_reader`
10+
```rust
11+
use std::io::{Seek, SeekFrom, Read};
12+
use std::fs::File;
13+
use deku::prelude::*;
14+
15+
#[derive(Debug, DekuRead, DekuWrite, PartialEq, Eq, Clone, Hash)]
16+
#[deku(endian = "big")]
17+
struct EcHdr {
18+
magic: [u8; 4],
19+
version: u8,
20+
padding1: [u8; 3],
21+
}
22+
23+
let mut file = File::options().read(true).open("file").unwrap();
24+
let ec = EcHdr::from_reader((&mut file, 0)).unwrap();
25+
```
26+
27+
- The more internal (with context) `read(..)` was replaced with `from_reader_with_ctx(..)`.
28+
With the switch to internal streaming, the variables `deku::input`, `deku::input_bits`, and `deku::rest` are now not possible and were removed.
29+
`deku::reader` is a replacement for some of the functionality.
30+
See [examples/deku_input.rs](examples/deku_input.rs) for a new example of caching all reads.
31+
32+
old:
33+
```rust
34+
#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
35+
struct DekuTest {
36+
field_a: u8,
37+
38+
#[deku(
39+
reader = "bit_flipper_read(*field_a, deku::rest, BitSize(8))",
40+
)]
41+
field_b: u8,
42+
}
43+
44+
fn custom_read(
45+
field_a: u8,
46+
rest: &BitSlice<u8, Msb0>,
47+
bit_size: BitSize,
48+
) -> Result<(&BitSlice<u8, Msb0>, u8), DekuError> {
49+
50+
// read field_b, calling original func
51+
let (rest, value) = u8::read(rest, bit_size)?;
52+
53+
Ok((rest, value))
54+
}
55+
```
56+
57+
new:
58+
```rust
59+
#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
60+
struct DekuTest {
61+
field_a: u8,
62+
63+
#[deku(
64+
reader = "bit_flipper_read(*field_a, deku::reader, BitSize(8))",
65+
)]
66+
field_b: u8,
67+
}
68+
69+
fn custom_read<R: std::io::Read>(
70+
field_a: u8,
71+
reader: &mut Reader<R>,
72+
bit_size: BitSize,
73+
) -> Result<u8, DekuError> {
74+
75+
// read field_b, calling original func
76+
let value = u8::from_reader_with_ctx(reader, bit_size)?;
77+
78+
Ok(value)
79+
}
80+
```
81+
82+
- With the addition of using `Read`, containing a byte slice with a reference is not supported:
83+
84+
old
85+
```rust
86+
#[derive(PartialEq, Debug, DekuRead, DekuWrite)]
87+
struct TestStruct<'a> {
88+
bytes: u8,
89+
90+
#[deku(bytes_read = "bytes")]
91+
data: &'a [u8],
92+
}
93+
```
94+
95+
new
96+
```rust
97+
#[derive(PartialEq, Debug, DekuRead, DekuWrite)]
98+
struct TestStruct {
99+
bytes: u8,
100+
101+
#[deku(bytes_read = "bytes")]
102+
data: Vec<u8>,
103+
}
104+
```
105+
106+
- `id_pat` is now required to be the same type as stored id.
107+
This also disallows using tuples for storing the id:
108+
109+
old:
110+
```rust
111+
#[derive(PartialEq, Debug, DekuRead, DekuWrite)]
112+
#[deku(type = "u8")]
113+
enum DekuTest {
114+
#[deku(id_pat = "_")]
115+
VariantC((u8, u8)),
116+
}
117+
```
118+
119+
new:
120+
```rust
121+
#[derive(PartialEq, Debug, DekuRead, DekuWrite)]
122+
#[deku(type = "u8")]
123+
enum DekuTest {
124+
#[deku(id_pat = "_")]
125+
VariantC {
126+
id: u8,
127+
other: u8,
128+
},
129+
}
130+
```
131+
132+
- The feature `const_generics` was removed and is enabled by default.
133+
5134
## [0.16.0] - 2023-02-28
6135

7136
### Changes

Cargo.toml

+4-3
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ keywords = ["deku", "bits", "serialization", "deserialization", "struct"]
99
categories = ["encoding", "parsing", "no-std"]
1010
description = "bit level serialization/deserialization proc-macro for structs"
1111
readme = "README.md"
12+
rust-version = "1.65.0"
1213

1314
[lib]
1415
bench = false
@@ -19,16 +20,16 @@ members = [
1920
]
2021

2122
[features]
22-
default = ["std", "const_generics"]
23-
std = ["deku_derive/std", "bitvec/std", "alloc"]
23+
default = ["std"]
24+
std = ["deku_derive/std", "bitvec/std", "alloc", "no_std_io/std"]
2425
alloc = ["bitvec/alloc"]
2526
logging = ["deku_derive/logging", "log"]
26-
const_generics = []
2727

2828
[dependencies]
2929
deku_derive = { version = "^0.16.0", path = "deku-derive", default-features = false}
3030
bitvec = { version = "1.0.1", default-features = false }
3131
log = { version = "0.4.17", optional = true }
32+
no_std_io = { version = "0.5.0", default-features = false, features = ["alloc"] }
3233

3334
[dev-dependencies]
3435
rstest = "0.16.0"

0 commit comments

Comments
 (0)