Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test #142

Merged
merged 27 commits into from
Nov 4, 2024
Merged

Fix test #142

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a521d0a
fix test
kp-cat Sep 20, 2024
43493e6
comment out block test
kp-cat Sep 20, 2024
afafa25
remove commented block
kp-cat Sep 20, 2024
8bc5f11
async: false test comment
kp-cat Sep 20, 2024
6d21c8d
comment out block test
kp-cat Sep 20, 2024
e69992c
test trace
kp-cat Sep 20, 2024
da24d76
validate sdk integration test turned off
kp-cat Sep 20, 2024
14fb99d
remove trace from github CI
kp-cat Sep 20, 2024
17f99b3
block test is back
kp-cat Sep 20, 2024
18d442d
- block test + test trace
kp-cat Sep 20, 2024
9430643
print Registry entry
kp-cat Sep 23, 2024
78c48ac
add registry items to warning message
kp-cat Sep 23, 2024
9784b11
message fix
kp-cat Sep 23, 2024
0b7fcd6
test finished log
kp-cat Sep 23, 2024
361bcaf
allow IO.puts
kp-cat Sep 23, 2024
03938e6
print registry
kp-cat Sep 23, 2024
fff2c50
clear registry before all test
kp-cat Sep 23, 2024
67ebd2c
add extra print_registry_contents
kp-cat Sep 24, 2024
f85f685
cleanup
kp-cat Sep 24, 2024
716f9c3
move block test into integration test
kp-cat Sep 24, 2024
20e1260
fix module in integration_test
kp-cat Oct 16, 2024
283b421
revert clear_registry + inline using_block_test
kp-cat Oct 29, 2024
d093a1e
run CI tests with max-cases 1
kp-cat Oct 30, 2024
b7dd1c4
run CI integration tests with max-cases 1
kp-cat Oct 30, 2024
5b76423
CI: merge coverall reports
kp-cat Oct 30, 2024
877ed28
remove max-cases 1
kp-cat Oct 31, 2024
1e1716c
Revert "remove max-cases 1"
kp-cat Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion test/config_cat_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
defmodule ConfigCatTest do
use ConfigCat.ClientCase, async: true
# Must be async: false to avoid a collision with other tests.
# Now that we only allow a single ConfigCat instance to use the same SDK key,
# one of the async tests would fail due to the existing running instance.
use ConfigCat.ClientCase, async: false

import ExUnit.CaptureLog
import Jason.Sigil
Expand Down
24 changes: 24 additions & 0 deletions test/integration_test.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ defmodule ConfigCat.IntegrationTest do

@sdk_key "configcat-sdk-1/PKDVCLf-Hq-h-kCzMp-L7Q/1cGEJXUwYUGZCBOL-E2sOw"

setup :clear_registry

describe "SDK key validation" do
test "raises error if SDK key is missing" do
nil
Expand Down Expand Up @@ -159,6 +161,20 @@ defmodule ConfigCat.IntegrationTest do
"default value"
end

defmodule CustomModule do
@moduledoc false
use ConfigCat, sdk_key: "configcat-sdk-1/PKDVCLf-Hq-h-kCzMp-L7Q/1cGEJXUwYUGZCBOL-E2sOw"
end

test "can call API through using block" do
_pid = start_supervised!(CustomModule)

:ok = CustomModule.force_refresh()

assert CustomModule.get_value("keySampleText", "default value") ==
"This text came from ConfigCat"
end

defp start(sdk_key, options \\ []) do
sdk_key
|> Cache.generate_key()
Expand All @@ -179,4 +195,12 @@ defmodule ConfigCat.IntegrationTest do

assert %ArgumentError{message: "SDK Key is required"} = error
end

defp clear_registry(_context) do
ConfigCat.Registry
|> Registry.match({__MODULE__, :_}, :_)
|> Enum.each(fn {key, _} -> Registry.unregister(ConfigCat.Registry, key) end)

:ok
end
Copy link
Collaborator

@randycoulman randycoulman Sep 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this part strictly necessary? Processes should remove themselves from the Registry when they shut down, and the use of start_supervised! ensures that processes are shut down when the test ends.

If it is necessary, are you sure that this code is doing what you want? __MODULE__ in the match spec will refer to this test's module (ConfigCat.IntegrationTest), but all of the started processes will have the module as ConfigCat.Supervisor, if I'm remembering correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first attempt was to simply turn off async test execution in ConfigCatTest. But on GitHub I still got the error. After that, I implemented the clear_registry function to ensure that the previous ConfigCat client instance is removed from the Registry. If I remember correctly, even with these changes, I occasionally encountered the error (though less frequently). Finally, I combined the two separate tests, and the error disappeared.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure that clear_registry isn't doing anything (because __MODULE__ is different in the test than it is in the code), but if this is working for you, feel free to merge it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right @randycoulman, I also got the error with the combined version (less frequently, but I think it is just because of the timing).

How should I use start_supervised! to ensure the process is shut down when the test ends?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can see, we're already using start_supervised in start_config_cat, which is what every test calls. Clearly something isn't shutting down fast enough, though. I have a different idea to try. I'll open up a draft PR once I have it working.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kp-cat See #143 for an alternate solution for this test problem.

If we find that the tests continue spuriously failing, we could change our CI config to run mix coveralls.json --warnings-as-errors --max-cases 1. That would force the tests to all run serially. That'll make the build a bit slower, but should make the problem go away.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @randycoulman,
I would avoid extra dependencies just because of this issue. I prefer the --max-cases 1 solution. I changed the CI config slightly to run the integration tests separately. What do you think about these changes here?

end
20 changes: 0 additions & 20 deletions test/using_block_test.exs
View file
Open in desktop

This file was deleted.

Loading