Should event sources and components be tied one to one?

[Seth Buntin]

I've been working through some ways to push deployment events from our Jenkins service to Compass.  I've worked through many different questions here and found the right mix to get the data pushing as I would expect it.  Once I created the event source for Jenkins I thought I could then attach that to multiple components and then start sending deployment events for each component.  What I am realizing is that the attaching of a component will make the deployment event show up on all the components attached to the event source since we aren't sending the component identifier in the event.  Does that sound right or am I mixing something up?  I find it less than ideal to make an event source from the same system for each component we might have building and deploying from Jenkins but since we aren't able to specify the component in the deployment event then maybe that is what we are forced to do?

[Josh Campbell · Accepted solution]

Hey Seth, you're right you need to create an event source for each of your components.  The basic challenge is ensuring the events are related to the correct component so our current model forces you to explicitly declare this.

Would it be helpful if we had a REST API to send events from a particular source but instead of declaring an externalSourceID/metricSource you'd just pass in a component ARI (id) or the name of the component you want to "receive" the event? Or what if you could send us a webhook from Jenkins and we could do something like parse out a tag/label or other property that would clue Compass into which component the event should belong to?

Could you share more about how you imagine setting this up in an ideal way, especially as it pertains to Jenkins?

[Seth Buntin]

Thanks!

Yeah, my initial thought was that I would attach an external source to a component and didn't connect that the event didn't have an attribute for a component.  I think, ideally, I would probably want to have one event source (this is coming from Jenkins) and the REST API for events take both the event source (where it's coming from) and the component (what is generating the event).

In the end, I don't mind using it this way; I want to make sure I understand the abstraction.  I think this particular abstraction, for something that isn't quite so "bootstrapped" for you, requires a decent understanding of the architecture so onboarding into Compass could be considered a bit heavy if you aren't aligned with any of the pre-built mechanisms.  Once it is done though I feel it fairly logical to maintain easily.  I've found within Jenkins that any form of pre-built solution is hit or miss since they are, almost always, dependent on an organization using it in that particular way.

If I were to choose the "perfect way for us" for Jenkins I would probably prefer a more event-driven approach for when the actions occurred instead of "one endpoint to rule them all".  We would start the build or deployment, and in that process we'd have an easy way to communicate via the build steps or pipeline.  The events would, potentially, look as such:

{eventSourceId: "jenkins", component: "project 1", build: {build_identifier: 1, started: <datetime>, ...}}

{eventSourceId: "jenkins", component: "project 3", deploy: {deploy_identifier: 2, started: <datetime>, ...}}

{eventSourceId: "jenkins", component: "project 3", deploy: {deploy_identifier: 2, finished: <datetime>, status: "SUCCESSFUL"}}

{eventSourceId: "jenkins", component: "project 1", build: {build_identifier: 1, finished: <datetime>, status: "FAILURE"}}

This pattern would mean that you all would have to derive the build and deploy times based on the build/deploy identifiers.  It would also mean that we might not need to do the whole "attach the datasource to the component" step as the event itself could assume it was attached.  The particular issue I'm running into at the moment is that I'm having to "passthrough" a ton of information from step to step to be sure I have everything I need at the end to be able to successfully send information to the current event endpoint.  Being able to send it when the event actually occurs makes it tons easier as those events can live in various different places within a pipeline.

[Josh Campbell]

Amazing, thanks for all the detail especially around the desired Jenkins scenario! Sharing this back with the team, super helpful.

I'm glad the way it works today makes sense, please let me know if I can help answer anything else :)

[Chase]

I'm with Seth on this. We shouldn't have to create an event source multiple times. It would be nice to just send a message to compass saying "this component started building", "this component finished building", "this component deployed", etc., without having to do all the tedious set up for every new component. As it is, we have to manage and track sequence numbers, source IDs, states, and dates for every component we want to integrate.