topic-manager improvements

[frairon]

This PR fixes two main issues:

• when checking if a topic exists and the cluster was configured with auto.create.topics.enable=true, the topic would be auto-created with default-configuration. This messes up table-creation because those topics would not be log-compacted which is required by goka. The new version does not trigger auto-creation when checking for a topic.
• if the topic configuration does not match, the topic manager can be configured to
  • print a warning
  • fail
  • ignore (that was the default before)
    The topic manager does not attempt to modify misconfigured topics, as this could be dangerous for intentional configuration changes via other kafka tools outside of goka.

[jomaresch]

LGTM

[R053NR07]

Think this looks good.

[R053NR07]

LGTM

[vladlepbeat]

Hi @frairon,
We updated to goka 1.0.6 recently and noticed this is a breaking change for us when spinning up testing environments. To explain a bit the use case, we use goka for a data pipeline and all the data we collect is deleted after 15 minutes. Although in goka compaction should be used, in our case it is actually not needed as everything is removed in 15 minutes. In those 15 mins compaction would not kick in anyways. We have testing environments that we can spin up with their own kafka cluster when needed. For these we have auto.create.topics.enable=true and in the past topics where created and all was working ok for us.

To fix the issue, we now downgraded to an older version but wanted to ask if we could have this configurable?

[jomaresch]

@vladlepbeat
As far as I know you can call EnsureStreamExists() on every startup for each topic you need.

Here is an example:

goka/examples/1-simplest/main.go

Lines 115 to 118 in 0085fa7

	err = tm.EnsureStreamExists(string(topic), 8)
	if err != nil {
	log.Printf("Error creating kafka topic %s: %v", topic, err)
	}

[vladlepbeat]

Hi @jomaresch
I tried now your proposal and it seems to do exactly what we need.
Thank you for the answer.

[jomaresch]

Nice.
Your are welcome 🙂

[frairon]

@vladlepbeat the thing with automatic topic creation was, that it was actually a bug in goka. We used the wrong API call so to say, which made clusters configured with auto-create create the topics also for tables. Then we ended up with table topics configured as streams (no compaction), which completely messes up the state in the views.
There was a similar discussion going on in issue #323 (which isn't solved yet).
If remembering correctly the bottom line is, that we want to implement a hybrid approach that makes goka manage the topics it owns (like tables, loops and output topics) and let the cluster decide for the other topics. So there's still a bit of work needed here. But Johannes' suggestion is the perfect workaround so far.