feat: New tedge http command

[didier-wenzek]

Proposed changes

Introduce a new tedge http sub command to interact with the c8y proxy, the entity store and the file transfer service.

This command uses tedge config to get the appropriate host, port and credentials to reach the c8y proxy or the agent depending on the requested uri. So the same command can be used unchanged from the main device or a child device, with TLS or even mTLS enabled or not.

Examples:

• tedge http get /c8y/inventory/managedObjects
• tedge http get /tedge/entity-store/v1/entities
• tedge http post /tedge/entity-store/v1/entities '{"@topic-id": "device/y//", "@type": "child-device", "@parent": "device/main//"}'

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s)
• I ran cargo fmt as mentioned in CODING_GUIDELINES
• I used cargo clippy as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 205 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/core/tedge/src/cli/http/cli.rs	0.00%	111 Missing ⚠️
crates/core/tedge/src/cli/http/command.rs	0.00%	93 Missing ⚠️
crates/core/tedge/src/cli/mod.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

📢 Thoughts on this report? Let us know!

[reubenmiller]

@didier-wenzek I'm wondering if we should create an tedge entity subcommand rather than tedge http.

Though having a command to talk to the c8y proxy might be useful, but the command would then have to pick out a different target server to send the HTTP request to (as the endpoint is owned by the mapper and not the tedge-agent) (you already mentioned this point, I just didn't read the full description before commenting)

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
577	0	3	577	100	1h35m29.391381s

[didier-wenzek]

There are no settings for c8y.proxy.client.auth. Am I missing something?

[reubenmiller]

Yeah you're right, that is a bit unexpected. Maybe the tedge-agent uses the http.client.auth.* properties when interacting with with the file transfer server or the c8y proxy (as the property is called http.client.auth*), but still it would open up for misconfiguration if the user configured the c8y.proxy (server) to use a different certificate as the tedge-agent's file transfer service (e.g. when they are not sharing certificates).

[reubenmiller]

I've created a ticket to address this part in a followup PR. #3388

[albinsuresh]

A really convenient addition to our cli toolset, as using curl for testing the HTTP APIs haven't been that smooth. Although I haven't done any extensive testing, esp the with proxy service, the basics seem good.

[albinsuresh]

Why not pass the Accept header also here?

[didier-wenzek]

Because, usually no content is expected from a PUT beyond a status.

[reubenmiller]

The file transfer service may not return anything, however the Cumulocity API can. This would also align with the tedge http post command.

So it is expected that the following should work:

# Create a managed object (so we can updated it later with http put)
MO_ID=$(tedge http post /c8y/inventory/managedObjects '{"name":"test"}' --accept-type application/json | jq -r '.id')

# Update an managed object (using the id created in the previous command)
tedge http put "/c8y/inventory/managedObjects/$MO_ID" '{"name":"item A"}' --accept-type application/json

Though it is fine to not have a default of the --content-type so that the Content-Type HTTP header will only be added to the request if the --content-type flag has a non-empty value.

[jarhodes314]

Do we also want to send a Content-Length header? I'm not sure how much attention Cumulocity will pay to this, and the file transfer service currently ignores this information (though I did create #3370 to improve this), but without it the server has absolutely no chance to establish if it received the entire payload or not.

[didier-wenzek]

Fixed by 5522d19

• Sending Content-Length
• Sending Accept on PUT (despite I do think this is a bad practice to respond with some content on PUT)
• Making Content-Type and Accept optional
• Guessing Content-Type from --file unless explicitly given with --content-type.

[reubenmiller]

I can confirm that the --content-type flag is now accepted, but is there anyway to show what MIME type was guessed? The --log-level trace shows very little useful information.

$ tedge http put /tedge/file-transfer/target.txt --file ./bootstrap.sh --log-level trace

2025-02-07T15:49:04.749797494Z DEBUG tedge_config::tedge_config_cli::tedge_config_location: Loading configuration from "/etc/tedge/tedge.toml"
2025-02-07T15:49:04.76825298Z TRACE reqwest::blocking::wait: (ThreadId(1)) park without timeout    
2025-02-07T15:49:04.768364731Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE    
2025-02-07T15:49:04.768392897Z TRACE reqwest::blocking::client: (ThreadId(2)) start runtime::block_on    
2025-02-07T15:49:04.771942794Z TRACE reqwest::blocking::wait: wait at most 30s    
2025-02-07T15:49:04.772032378Z TRACE reqwest::blocking::wait: (ThreadId(1)) park timeout 29.999934624s    
2025-02-07T15:49:04.772057545Z TRACE hyper::client::pool: checkout waiting for idle connection: ("http", 127.0.0.1:8000)
2025-02-07T15:49:04.77208892Z DEBUG reqwest::connect: starting new connection: http://127.0.0.1:8000/    
2025-02-07T15:49:04.772110045Z TRACE hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("127.0.0.1"), port=Some(Port(8000))
2025-02-07T15:49:04.772124753Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8000
2025-02-07T15:49:04.772289421Z TRACE mio::poll: registering event source with poller: token=Token(268779028827008), interests=READABLE | WRITABLE    
2025-02-07T15:49:04.772324796Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8000
2025-02-07T15:49:04.772352838Z TRACE hyper::client::conn: client handshake Http1
2025-02-07T15:49:04.772383922Z TRACE hyper::client::client: handshake complete, spawning background dispatcher task
2025-02-07T15:49:04.772434714Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2025-02-07T15:49:04.772474547Z TRACE hyper::client::pool: checkout dropped for ("http", 127.0.0.1:8000)
2025-02-07T15:49:04.772526506Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=PUT, body=Some(Unknown)
2025-02-07T15:49:04.772570631Z TRACE hyper::proto::h1::encode: sized write, len = 8192
2025-02-07T15:49:04.772580298Z TRACE hyper::proto::h1::io: buffer.flatten self.len=140 buf.len=8192
2025-02-07T15:49:04.772581381Z TRACE reqwest::blocking::wait: (ThreadId(1)) park timeout 29.999385288s    
2025-02-07T15:49:04.772596881Z TRACE hyper::proto::h1::encode: sized write, len = 8192
2025-02-07T15:49:04.772601923Z TRACE hyper::proto::h1::io: buffer.flatten self.len=8332 buf.len=8192
2025-02-07T15:49:04.772620965Z TRACE reqwest::blocking::wait: (ThreadId(1)) park timeout 29.999344454s    
2025-02-07T15:49:04.772635923Z DEBUG hyper::proto::h1::io: flushed 16524 bytes
2025-02-07T15:49:04.77264509Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Body(Encoder { kind: Length(16249), is_last: false }), keep_alive: Busy }
2025-02-07T15:49:04.772650298Z TRACE hyper::proto::h1::conn: Conn::read_head
2025-02-07T15:49:04.77267859Z TRACE hyper::proto::h1::encode: sized write, len = 8192
2025-02-07T15:49:04.77268684Z TRACE hyper::proto::h1::io: buffer.flatten self.len=0 buf.len=8192
2025-02-07T15:49:04.772691048Z TRACE hyper::proto::h1::encode: sized write, len = 8057
2025-02-07T15:49:04.772697507Z TRACE hyper::proto::h1::io: buffer.flatten self.len=8192 buf.len=8057
2025-02-07T15:49:04.77271584Z TRACE hyper::proto::h1::dispatch: no more write body allowed, user body is_end_stream = false
2025-02-07T15:49:04.772750049Z DEBUG hyper::proto::h1::io: flushed 16249 bytes
2025-02-07T15:49:04.772702715Z TRACE reqwest::blocking::wait: (ThreadId(1)) park timeout 29.999263954s    
2025-02-07T15:49:04.772765132Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2025-02-07T15:49:04.772775174Z TRACE reqwest::blocking::wait: (ThreadId(1)) park timeout 29.999191203s    
2025-02-07T15:49:04.774305308Z TRACE hyper::proto::h1::conn: Conn::read_head
2025-02-07T15:49:04.7743286Z TRACE hyper::proto::h1::io: received 80 bytes
2025-02-07T15:49:04.774336183Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=80
2025-02-07T15:49:04.774341558Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(80)
2025-02-07T15:49:04.774350433Z DEBUG hyper::proto::h1::io: parsed 2 headers
2025-02-07T15:49:04.774352725Z DEBUG hyper::proto::h1::conn: incoming body is empty
2025-02-07T15:49:04.774356142Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2025-02-07T15:49:04.774362225Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2025-02-07T15:49:04.774372934Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2025-02-07T15:49:04.77439635Z TRACE hyper::client::pool: put; add idle connection for ("http", 127.0.0.1:8000)
2025-02-07T15:49:04.7744086Z DEBUG hyper::client::pool: pooling idle connection for ("http", 127.0.0.1:8000)
2025-02-07T15:49:04.774482643Z TRACE reqwest::blocking::wait: wait at most 30s    
2025-02-07T15:49:04.774506643Z TRACE reqwest::blocking::client: closing runtime thread (ThreadId(2))    
2025-02-07T15:49:04.774515184Z TRACE reqwest::blocking::client: signaled close for runtime thread (ThreadId(2))    
2025-02-07T15:49:04.77465331Z TRACE reqwest::blocking::client: (ThreadId(2)) Receiver is shutdown    
2025-02-07T15:49:04.775335398Z TRACE reqwest::blocking::client: (ThreadId(2)) end runtime::block_on    
2025-02-07T15:49:04.775349356Z TRACE mio::poll: deregistering event source from poller    
2025-02-07T15:49:04.775849942Z TRACE reqwest::blocking::client: (ThreadId(2)) finished    
2025-02-07T15:49:04.775951735Z TRACE reqwest::blocking::client: closed runtime thread (ThreadId(2))

[albinsuresh]

LGTM

[reubenmiller]

Approved. Really nice addition

[Bravo555]

No blockers from me, however I don't really understand the purpose of arg2 argument.

#3357 (comment)