Is there a universal interface I can use?

Discussion:

Tilman Baumann

2017-11-22 15:02:08 UTC

I'm writing a reactive subordinate charm for cassandra.
I can not find a interface for cassandra. But that's ok, since I don't
really need a full blown database connection client.

Easy I thought and just re-used the juju-info interface for fun and profit.
requires:
host-system:
interface: juju-info
scope: container
database:
interface: juju-info

And in the code
@when('database.available')
def db_changed(cassandra):
for conv in cassandra.conversations():
username = conv.get_remote('username')
password = conv.get_remote('password')
...

However, that doesn't seem to work. Juju complains the relation doesn't
exist.
$ juju add-relation cassandra-backup:database cassandra:database
ERROR no relations found

So, is there a interface that I can (ab-)use in a similar way?

I don't want to build a full blown cassandra interface and at it to the
list.

Cheers and thanks
Tilman

--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/ma

Cory Johns

2017-11-22 17:51:56 UTC

Permalink

You can use the "juju-info" interface type for subordinates, but I'm not
sure that this is the correct thing for your case.

You're trying to attach the non-subordinate "database" endpoint (of
interface protocol "juju-info") on your charm to the "database" endpoint
(of interface protocol "cassandra") on the cassandra charm. Because the
interface protocols don't match, Juju will not establish the relation.
Additionally, because you're not connecting to the subordinate endpoint
(scope: container) and instead to the standard endpoint, even if the
relation was established, you would never see an instance of your charm
created because it has no host application.

All charms have an implicit relation endpoint called "juju-info" of
interface protocol "juju-info" that you can connect to, and as long as the
interface protocol match and are unambiguous, you can omit them. To
connect your charm using the "juju-info" interface protocol, you would need
to only define the "host-system" endpoint in your charm (as you currently
have it), leave off the "database" endpoint, and omit the endpoint portion
when adding the relation (or, if you want to be explicit about that, you
can use "juju add-relation cassandra-backup:host-system
cassandra:juju-info"). Then, for the reactive portion, you would use
"@when('host-system.available')" as the decorator.

However, I see that you also want to retrieve relation data that the
cassandra charm provides using the "cassandra" interface protocol. Here it
becomes important to note that whether a relation is subordinate or not is
independent of the interface protocol the relation uses. Thus, you could
mark the endpoint as "scope: container" and still use the "cassandra"
interface protocol. You would then want to drop the "host-system" endpoint
from your charm, mark the "database" endpoint as "interface: cassandra",
and connect as you were trying to do. However, I seem to recall that
Stuart (the maintainer of the cassandra charm) raised a possible issue with
subordinates using relations that the other end doesn't expect to be
subordinate, so there might be some concerns there. I can't recall exactly
what the issue was, and I can't seem to find the bug report now. Perhaps
Stuart can chime in.

A final concern is that I don't see an interface layer for the "cassandra"
interface protocol in the layer index (
https://github.com/juju/layer-index#list-of-interface-layers), meaning it
will be more difficult to use in a reactive charm. Without the interface
layer, you wouldn't get the flags set to react to and would need to
manually inspect the relation data using charmhelpers.core.hookenv. It
looks like your use case is fairly simple, though, so we can probably get a
very basic interface layer for it put together pretty quickly that Stuart
could then expand upon. I'm happy to help with that, but with the holidays
in the US, I might be a few days before I can do so.

On Wed, Nov 22, 2017 at 10:02 AM, Tilman Baumann <

Post by Tilman Baumann
I'm writing a reactive subordinate charm for cassandra.
I can not find a interface for cassandra. But that's ok, since I don't
really need a full blown database connection client.
Easy I thought and just re-used the juju-info interface for fun and profit.
interface: juju-info
scope: container
interface: juju-info
And in the code
@when('database.available')
username = conv.get_remote('username')
password = conv.get_remote('password')
...
However, that doesn't seem to work. Juju complains the relation doesn't
exist.
$ juju add-relation cassandra-backup:database cassandra:database
ERROR no relations found
So, is there a interface that I can (ab-)use in a similar way?
I don't want to build a full blown cassandra interface and at it to the
list.
Cheers and thanks
Tilman
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju

Dmitrii Shcherbakov

2017-11-22 18:57:52 UTC

Permalink

Also:

"scope: global" vs "scope: container" in metadata.yaml are Juju-level
per-relation concepts.

Regardless of reactive or non-reactive charms, when you do

juju add-relation <primary-app>:<endpoint-name>
<subordinate-app>:<endpoint-name>

relation data will flow only between primary and subordinate units on a
given logical machine (machine or container). In that sense "scope:
container" means primary <-> subordinate on the same "logical machine"
(doesn't have to be an LXD container).

IoW:

scope: container
N relation "instances" between units, 2 * N relation lifecycles (one per
unit)
or a primary only talks to its subordinate over that relation.

scope: global
N * N relation "instances" between units, 2 * N^2 relation lifecycles (N
per unit)
a primary talks to a subordinate and any other same-app primary's
subordinate over that relation

Note that being primary or subordinate does not impact your ability to set
up "scope: global" relations - being a subordinate only affects your
placement and units you can observe over a "scope: container" relation.

Best Regards,
Dmitrii Shcherbakov

Field Software Engineer
IRC (freenode): Dmitrii-Sh

Post by Cory Johns
You can use the "juju-info" interface type for subordinates, but I'm not
sure that this is the correct thing for your case.
You're trying to attach the non-subordinate "database" endpoint (of
interface protocol "juju-info") on your charm to the "database" endpoint
(of interface protocol "cassandra") on the cassandra charm. Because the
interface protocols don't match, Juju will not establish the relation.
Additionally, because you're not connecting to the subordinate endpoint
(scope: container) and instead to the standard endpoint, even if the
relation was established, you would never see an instance of your charm
created because it has no host application.
All charms have an implicit relation endpoint called "juju-info" of
interface protocol "juju-info" that you can connect to, and as long as the
interface protocol match and are unambiguous, you can omit them. To
connect your charm using the "juju-info" interface protocol, you would need
to only define the "host-system" endpoint in your charm (as you currently
have it), leave off the "database" endpoint, and omit the endpoint portion
when adding the relation (or, if you want to be explicit about that, you
can use "juju add-relation cassandra-backup:host-system
cassandra:juju-info"). Then, for the reactive portion, you would use
However, I see that you also want to retrieve relation data that the
cassandra charm provides using the "cassandra" interface protocol. Here it
becomes important to note that whether a relation is subordinate or not is
independent of the interface protocol the relation uses. Thus, you could
mark the endpoint as "scope: container" and still use the "cassandra"
interface protocol. You would then want to drop the "host-system" endpoint
from your charm, mark the "database" endpoint as "interface: cassandra",
and connect as you were trying to do. However, I seem to recall that
Stuart (the maintainer of the cassandra charm) raised a possible issue with
subordinates using relations that the other end doesn't expect to be
subordinate, so there might be some concerns there. I can't recall exactly
what the issue was, and I can't seem to find the bug report now. Perhaps
Stuart can chime in.
A final concern is that I don't see an interface layer for the "cassandra"
interface protocol in the layer index (https://github.com/juju/
layer-index#list-of-interface-layers), meaning it will be more difficult
to use in a reactive charm. Without the interface layer, you wouldn't get
the flags set to react to and would need to manually inspect the relation
data using charmhelpers.core.hookenv. It looks like your use case is
fairly simple, though, so we can probably get a very basic interface layer
for it put together pretty quickly that Stuart could then expand upon. I'm
happy to help with that, but with the holidays in the US, I might be a few
days before I can do so.
On Wed, Nov 22, 2017 at 10:02 AM, Tilman Baumann <

--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju

Tilman Baumann

2017-11-24 14:19:32 UTC

Permalink

This post might be inappropriate. Click to display it.

Haw Loeung

2017-11-22 22:26:22 UTC

Permalink

Hi Tilman,

Post by Tilman Baumann
However, that doesn't seem to work. Juju complains the relation doesn't
exist.
$ juju add-relation cassandra-backup:database cassandra:database
ERROR no relations found
So, is there a interface that I can (ab-)use in a similar way?
I don't want to build a full blown cassandra interface and at it to the
list.

Not sure if you've seen this, but I did some work recently with
something similar to backup Cassandra DBs:

| https://jujucharms.com/u/hloeung/cassandra-backup/

It's currently still experimental and uses the CQL COPY
TO... commands. The Cassandra charm already ships out the required
credentials so tools such as cqlsh should just work.

Regards,

Haw

John Meinel

2017-11-23 03:01:33 UTC

Permalink

I did start working on a Cassandra interface for something I was working
on. I don't know that it is complete but
https://github.com/jameinel/interface-cassandra

Was my attempt at it.

John
=:->

Post by Haw Loeung
Hi Tilman,

Not sure if you've seen this, but I did some work recently with
| https://jujucharms.com/u/hloeung/cassandra-backup/
It's currently still experimental and uses the CQL COPY
TO... commands. The Cassandra charm already ships out the required
credentials so tools such as cqlsh should just work.
Regards,
Haw
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju

Tilman Baumann

2017-11-23 10:39:36 UTC

Permalink

Cool. Thanks

The two fields I was interested in, username and password are mising
though. :D
But I'm thinking right now if I even want to go that route...

Post by John Meinel
I did start working on a Cassandra interface for something I was working
on. I don't know that it is complete but
https://github.com/jameinel/interface-cassandra
Was my attempt at it.
John
=:->
Hi Tilman,

Post by Tilman Baumann
However, that doesn't seem to work. Juju complains the relation

doesn't

Post by Tilman Baumann
exist.
$ juju add-relation cassandra-backup:database cassandra:database
ERROR no relations found
So, is there a interface that I can (ab-)use in a similar way?
I don't want to build a full blown cassandra interface and at it

to the

Post by Tilman Baumann
list.

Not sure if you've seen this, but I did some work recently with
| https://jujucharms.com/u/hloeung/cassandra-backup/
<https://jujucharms.com/u/hloeung/cassandra-backup/>
It's currently still experimental and uses the CQL COPY
TO... commands. The Cassandra charm already ships out the required
credentials so tools such as cqlsh should just work.
Regards,
Haw
--
Juju mailing list
https://lists.ubuntu.com/mailman/listinfo/juju
<https://lists.ubuntu.com/mailman/listinfo/juju>

--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe at: https://

Tilman Baumann

2017-11-24 14:24:38 UTC

Permalink

Hey John, apart from those fields missing. (BTW check usage section of
https://jujucharms.com/cassandra/36 for a list of fields)

interface.yaml calls this interface elasticsearch. I was able to use
this after I changed that.

;-)

Cheers
Tilman

Post by Tilman Baumann
Cool. Thanks
The two fields I was interested in, username and password are mising
though. :D
But I'm thinking right now if I even want to go that route...

--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe a

John Meinel

2017-11-26 12:56:45 UTC

Permalink

My apologies. I had done a bunch of local updates and had not yet pushed
them back to github.

There should be quite a bit of changes, including a lot more of the values
and changing interfaces.yaml, etc.

John
=:->

On Fri, Nov 24, 2017 at 6:24 PM, Tilman Baumann <

Post by Tilman Baumann
Hey John, apart from those fields missing. (BTW check usage section of
https://jujucharms.com/cassandra/36 for a list of fields)
interface.yaml calls this interface elasticsearch. I was able to use
this after I changed that.
;-)
Cheers
Tilman

Post by Tilman Baumann
Cool. Thanks
The two fields I was interested in, username and password are mising
though. :D
But I'm thinking right now if I even want to go that route...

--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju

Tilman Baumann

2017-11-23 10:37:52 UTC

Permalink

Post by Haw Loeung
Hi Tilman,

Not sure if you've seen this, but I did some work recently with
| https://jujucharms.com/u/hloeung/cassandra-backup/

I didn't want to talk about it before it's usable. I think I might be
working on something similar.

https://github.com/tbaumann/jujucharm-layer-cassandra-backup

It seems to only use "nodetool snapshot"
I'm integrating this for a 3rd party so I don't quite know what is going
on there. But looks like the intent is pretty much the same.

Post by Haw Loeung
It's currently still experimental and uses the CQL COPY
TO... commands. The Cassandra charm already ships out the required
credentials so tools such as cqlsh should just work.

Good point. And worst case I can read the yaml file for cassandra
myself. Saves me a cumbersome additional relation.
I was undecided about this. But this is shaping up to be the cleaner
solution...

Stuart Bishop

2017-11-27 23:38:11 UTC

Permalink

On 23 November 2017 at 21:37, Tilman Baumann

Post by Tilman Baumann

Post by Haw Loeung
Hi Tilman,

Not sure if you've seen this, but I did some work recently with
| https://jujucharms.com/u/hloeung/cassandra-backup/

I didn't want to talk about it before it's usable. I think I might be
working on something similar.
https://github.com/tbaumann/jujucharm-layer-cassandra-backup
It seems to only use "nodetool snapshot"
I'm integrating this for a 3rd party so I don't quite know what is going
on there. But looks like the intent is pretty much the same.

I think this charm needs to remain a subordinate, because 'nodetool
snapshot' requires a JMX connection and that should be blocked
(because it is not secured).

I'd be happy to have actions on the main Cassandra charm to manage
snapshots, and cronned snapshots would also be a feature suitable for
the main charm. But you would still need some way to ship the
snapshots to your backup host which should be a subordinate.

Ideally the Cassandra charm would support multiple DCs, which would
allow you to backup only a subset of your nodes to get a complete copy
of your data, but that is going to need to wait until a
charms.reactive rewrite.

--
Stuart Bishop <***@canonical.com>
--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/

Tilman Baumann

2017-11-28 09:22:28 UTC

Permalink

Post by Stuart Bishop
On 23 November 2017 at 21:37, Tilman Baumann

Post by Tilman Baumann
I didn't want to talk about it before it's usable. I think I might be
working on something similar.
https://github.com/tbaumann/jujucharm-layer-cassandra-backup
It seems to only use "nodetool snapshot"
I'm integrating this for a 3rd party so I don't quite know what is going
on there. But looks like the intent is pretty much the same.

That would be quite nice actually. Backup and snapshot could be two
different actions even.
Snapshot is a little low-level as it is per-node. But it makes for fast
recovery if a node hickups.
Full backup like Haw Leoeung implemented in
https://jujucharms.com/u/hloeung/cassandra-backup/ is probably even more
useful for many scenarios.

I'm quite stumped right now with the odd combination of needing to be a
subordinate and needing to connect to the database relation.
I just can't get it to work. And I'm out of ideas.
I would love to just finish it "to get it to work" but I don't know what
else I could try at this point.

Generally I like the idea of just plonking on subordinates to extend
features.
But backup is a quite central one. Makes sense in the main charm.

The percona charm has for example a backup action too. I just checked.
It also just places a file in the file-system.
(No cron scheduled one. But that would be useful there too)

Very nice would be a unified backup plugin thing that lives in
charmhelpers and could interact with some backup server. Wishful
thinking... :D
Might eve be a nice use of the not yet supported shared storage thing
https://jujucharms.com/docs/stable/charms-storage

I'm not very familiar with the coding style used in the cassandra charm.
But I think I could help you with adding those functionalities even.
I will have a lot of distractions the next two weeks. But I can see what
I can do...

Thanks
Tilman

--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/lis

Stuart Bishop

2017-12-01 05:17:35 UTC

Permalink

On 28 November 2017 at 20:22, Tilman Baumann

Post by Tilman Baumann
That would be quite nice actually. Backup and snapshot could be two
different actions even.
Snapshot is a little low-level as it is per-node. But it makes for fast
recovery if a node hickups.
Full backup like Haw Leoeung implemented in
https://jujucharms.com/u/hloeung/cassandra-backup/ is probably even more
useful for many scenarios.

There doesn't seem to be a standard backup mechanism, so backup
probably belongs in a subordinate. I currently have a subordinate that
rsyncs the files to a backup host on my smallest deployment (per
https://insights.ubuntu.com/2015/08/04/introducing-turku-cloud-friendly-backups-for-your-infrastructure/)

Post by Tilman Baumann
I'm quite stumped right now with the odd combination of needing to be a
subordinate and needing to connect to the database relation.
I just can't get it to work. And I'm out of ideas.
I would love to just finish it "to get it to work" but I don't know what
else I could try at this point.

You just need to define the juju-info container scoped relation as
well as the standard cassandra relation.

subordinate: true
requires:
cassandra:
interface: cassandra
juju-info:
interface: juju-info
scope: container

When you deploy, you need to connect both relations and you will have
them both become available, with the different scopes:

juju add-relation mysub:juju-info cassandra:juju-info
juju add-relation mysub:cassandra cassandra:database

Post by Tilman Baumann
I'm not very familiar with the coding style used in the cassandra charm.
But I think I could help you with adding those functionalities even.
I will have a lot of distractions the next two weeks. But I can see what
I can do...

I want to make time to rework it as a charms.reactive charm. But
adding an action shouldn't need that, as the actions.yaml and
actions/foo script don't need any dependencies on the rest of the
charm.

--
Stuart Bishop <***@canonical.com>
--
Juju mailing list
***@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm

Tilman Baumann

2017-11-24 13:15:39 UTC

Permalink

Post by Haw Loeung
Hi Tilman,

In light of this, I should perhaps consider renaming my charm
cassandra-snapshots.
End goal might be to combine the two.