Mock s3fs testing framework #26
Conversation
tests/test_mock_s3fs.py
Outdated
| os.environ["AWS_SECRET_ACCESS_KEY"] = "foo" | ||
| if "AWS_ACCESS_KEY_ID" not in os.environ: | ||
| os.environ["AWS_ACCESS_KEY_ID"] = "foo" | ||
| os.environ.pop("AWS_PROFILE", None) |
There was a problem hiding this comment.
What's going on with the credentials here? Is the objective to create credentials which only have scope within the tests? If so, I think it would be worth being explicit in comments, both here where the credentials are created, and where the (test scoped) credentials are being used.
There was a problem hiding this comment.
hang on, good call, let me add some inline comments here
tests/test_mock_s3fs.py
Outdated
| client.create_bucket(Bucket=versioned_bucket_name, ACL="public-read") | ||
| client.put_bucket_versioning( | ||
| Bucket=versioned_bucket_name, VersioningConfiguration={"Status": "Enabled"} | ||
| ) |
There was a problem hiding this comment.
Does Public Read mean the credentials created earlier are ignored?
There was a problem hiding this comment.
public is NOT anon, this is like your bucket on CEDA/ACES
|
Could you please add another test (in another file) which actually uses this fixture, that we can then clone/copy/extract bits of, for the real tests? |
| endpoint_uri = "http://127.0.0.1:%s/" % port | ||
|
|
||
|
|
||
| def test_s3fs_s3(s3fs_s3): |
There was a problem hiding this comment.
@bnlawrence here is a very little test that sets up the mock S3 FS via its fixture in conftest - the object mock_s3_filesystem is a true S3 FS with all its attributes and methods, and you can use it to do puts and gets and what nots. It's just virtual, not a real thing 😁
| s3 = s3fs.S3FileSystem(anon=False, client_kwargs={"endpoint_url": endpoint_uri}) | ||
| s3.invalidate_cache() | ||
|
|
||
| yield s3 |
There was a problem hiding this comment.
conftest now added, gents @bnlawrence and @davidhassell - these fixtures are now available out the box for testing right away
|
I have had the following when I used the merge branch: which is likely to bite us on the bum in pyactivestorage? |
Sister PR to NCAS-CMS/PyActiveStorage#232
We mock an entire S3 fsspec/s3fs-like File System, pop a working HTTP client on top of it, and add a real POSIX file to it, then access it as if it was a real file but in an S3 s3fs-like FS. Wild Weasel YGBSM stuff 😁