Endpoints trials + MAAS charms

Discussion:

James Beedy

2017-11-26 01:14:53 UTC

Hello all,

I've got an HA maas setup at the datacenter, I had some trouble getting the
full HA bits super solid in the past, and thought it appropriate to try
charming up the new 2.3 MAAS snaps to see if I could make my life a bit
easier going forward.

I just took a quick first swipe at this, so please excuse the lack of any
tests.

I'm hoping I can kill 2 birds with 1 stone here by a) possibly getting some
feedback from @cory_fu on how I'm using the new Endpoints stuff landing
soon in reactive (and disseminate that feedback so others will be in the
know too), and b) possibly someone from @MAAS team might give me a nod if
the steps I've taken here look to be moving in the right direction?

# Interface and layer
interface-maas: https://github.com/jamesbeedy/interface-maas
layer-maas: https://github.com/jamesbeedy/layer-maas

# MAAS charm
charmstore: https://jujucharms.com/u/jamesbeedy/maas/8

# Sample bundle
sample bundle: http://paste.ubuntu.com/26046016/ - (only for reference,
this won't actually deploy)

# juju status
`juju status`: http://paste.ubuntu.com/26045880/

Thanks,

James

James Beedy

2017-11-27 13:17:09 UTC

Permalink

Dmitrii,

Thanks for the response.

When taking into account the overhead to get MAAS deployed as charms, it
definitely makes the LXD provider method you have described seem very
appealing. Some issues I've experienced trying to get MAAS HA deployed are
such that it seems like just a ton of infrastructure is needed to get MAAS
HA standing using the manual provider deploying the MAAS charms. You have
to provision/maintain the manual Juju controller cluster underneath MAAS,
just to have MAAS .... ugh

I found not only the sheer quantity of servers needed to get this working
quite unnerving, but also the manual ops I had to undergo to get it all
standing #snowflakes #custom.

I iterated on this a few times to see if I could come up with something
more manageable, and this is where I landed (brace yourself) [0]

What's going on there?

I created a model in JAAS, manually added 3 physical hosts across different
racks, then bootstrapped 4 virtual machines on each physical host (1 vm for
each postgresql, maas-region, maas-rack [1], juju-controller (juju
controllers for maas provider, to be checked into maas)) on each host.

I then also added my vms to my JAAS model so I could deploy the charms to
them (just postgresql and ubuntu at the time - the MAAS stuff got manually
installed and configured after the machiens had ubuntu deployed to them in
the model).

(ASIDE: I'm seeing I could probably do this one better by combining my
region and rack controller to the same vm, and colocating the postgresql
charm with the region+rack on the same vm, giving me ha of all components
using single vm on each host.)

I know there are probably a plethora of issues with what I've done here,
but it solved a few primary issues that seemed to outweigh the misuse.

The issues were:

1) Too many machines needed to get MAAS HA
2) Chicken or egg?
3) Snowflakes!!!
4) No clear path to support MAAS HA

My idea behind this was such that by using JAAS I am solving the chicken or
the egg issue, and reducing the machines/infra needed to get MAAS HA.
Deploying the MAAS infra with Juju eliminates the snowflake/lack of
tracking and chips at the "No clear path to support MAAS HA".

Thanks,

James

[0] http://paste.ubuntu.com/25891429/
[1] http://paste.ubuntu.com/26058033/

On Mon, Nov 27, 2017 at 12:09 AM, Dmitrii Shcherbakov <

Hi James,
This is an interesting approach, thanks for taking a shot at solving this
problem!
I thought of doing something similar a few months ago. The problematic
aspect here is the assumption of having a provider/substrate already
present for MAAS to be deployed - this is the chicken or the egg type of
problem.
If you would like to take the MAAS charm route, manual provider could be
used with Juju to do that with pre-created hosts (which may be
containers/VMs/hosts all in a single model with this provider, regardless
of how they were deployed). There would be hosts for a Juju controller(s)
and MAAS region/rack controllers in the end.
If you put both Juju controller and MAAS into containers, it gives you
some flexibility. If you are careful, you can even migrate those
containers. Running MAAS in an unprivileged container should be perfectly
possible https://github.com/CanonicalLtd/maas-docs/issues/700 - I am not
sure that the instructions that require a privileged container with loop
devices passed to it are relevant anymore.
An alternative is to use the lxd provider (which can connect to a remote
daemon, not only localhost) but this is only one daemon per provider. For
HA purposes you would need several LXDs on different hosts and for this
provider to support network spaces because you may have MAAS hosts located
in different layer 2 networks with different subnets used. Cross-model
relations could be used to have a model per LXD provider but I am not sure
this is the best approach - units would be on different models with no
shared unit-level leadership.
https://github.com/juju/juju/tree/develop/provider/lxd
With the new LXD clustering work it might be possible overcome this
limitation as well. I would assume LXD clustering to work on a per-rack
basis due to latency constraints while with MAAS in a data center you would
surely place different region controllers and rack controllers on different
racks (availability zones).
https://insights.ubuntu.com/2017/10/23/lxd-weekly-status-
20-authentication-conferences-more/
"Distributed database for LXD clustering"
If, by the time of LXD clustering release, there was support for
availability zones it would have solved the problem with a single control
plane for a Juju provider in the absence of MAAS.
An alternative to the above is just usage of bootstrap automation to set
up MAAS and then usage of Juju with charms for the rest of what you need.
Best Regards,
Dmitrii Shcherbakov
Field Software Engineer
IRC (freenode): Dmitrii-Sh

Post by James Beedy
Hello all,
I've got an HA maas setup at the datacenter, I had some trouble getting
the full HA bits super solid in the past, and thought it appropriate to try
charming up the new 2.3 MAAS snaps to see if I could make my life a bit
easier going forward.
I just took a quick first swipe at this, so please excuse the lack of any
tests.
I'm hoping I can kill 2 birds with 1 stone here by a) possibly getting
landing soon in reactive (and disseminate that feedback so others will be
nod if the steps I've taken here look to be moving in the right direction?
# Interface and layer
interface-maas: https://github.com/jamesbeedy/interface-maas
layer-maas: https://github.com/jamesbeedy/layer-maas
# MAAS charm
charmstore: https://jujucharms.com/u/jamesbeedy/maas/8
# Sample bundle
sample bundle: http://paste.ubuntu.com/26046016/ - (only for reference,
this won't actually deploy)
# juju status
`juju status`: http://paste.ubuntu.com/26045880/
Thanks,
James
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju

John Meinel

2017-11-27 13:31:24 UTC

Permalink

It does seem like Juju operating the LXD provider, but spanning multiple
machines would be an answer. I do believe that LXD itself is introducing
clustering into their API in the 18.04 cycle. Which would probably need
some updates on the Juju side to handle their targeted provisioning (create
a container for maas-postgresql on the 3rd machine in the LXD cluster).
But that might get you out of manual provisioning of a bunch of machines
and VMs to target everything.
We're certainly not at a point where you could just-do-that today.
I can also see that it took a few attempts to get there (the last machine
is #40 :)
John
=:->

Post by James Beedy
Dmitrii,
Thanks for the response.
When taking into account the overhead to get MAAS deployed as charms, it
definitely makes the LXD provider method you have described seem very
appealing. Some issues I've experienced trying to get MAAS HA deployed are
such that it seems like just a ton of infrastructure is needed to get MAAS
HA standing using the manual provider deploying the MAAS charms. You have
to provision/maintain the manual Juju controller cluster underneath MAAS,
just to have MAAS .... ugh
I found not only the sheer quantity of servers needed to get this working
quite unnerving, but also the manual ops I had to undergo to get it all
standing #snowflakes #custom.
I iterated on this a few times to see if I could come up with something
more manageable, and this is where I landed (brace yourself) [0]
What's going on there?
I created a model in JAAS, manually added 3 physical hosts across
different racks, then bootstrapped 4 virtual machines on each physical host
(1 vm for each postgresql, maas-region, maas-rack [1], juju-controller
(juju controllers for maas provider, to be checked into maas)) on each host.
I then also added my vms to my JAAS model so I could deploy the charms to
them (just postgresql and ubuntu at the time - the MAAS stuff got manually
installed and configured after the machiens had ubuntu deployed to them in
the model).
(ASIDE: I'm seeing I could probably do this one better by combining my
region and rack controller to the same vm, and colocating the postgresql
charm with the region+rack on the same vm, giving me ha of all components
using single vm on each host.)
I know there are probably a plethora of issues with what I've done here,
but it solved a few primary issues that seemed to outweigh the misuse.
1) Too many machines needed to get MAAS HA
2) Chicken or egg?
3) Snowflakes!!!
4) No clear path to support MAAS HA
My idea behind this was such that by using JAAS I am solving the chicken
or the egg issue, and reducing the machines/infra needed to get MAAS HA.
Deploying the MAAS infra with Juju eliminates the snowflake/lack of
tracking and chips at the "No clear path to support MAAS HA".
Thanks,
James
[0] http://paste.ubuntu.com/25891429/
[1] http://paste.ubuntu.com/26058033/
On Mon, Nov 27, 2017 at 12:09 AM, Dmitrii Shcherbakov <

Hi James,
This is an interesting approach, thanks for taking a shot at solving this
problem!
I thought of doing something similar a few months ago. The problematic
aspect here is the assumption of having a provider/substrate already
present for MAAS to be deployed - this is the chicken or the egg type of
problem.
If you would like to take the MAAS charm route, manual provider could be
used with Juju to do that with pre-created hosts (which may be
containers/VMs/hosts all in a single model with this provider, regardless
of how they were deployed). There would be hosts for a Juju controller(s)
and MAAS region/rack controllers in the end.
If you put both Juju controller and MAAS into containers, it gives you
some flexibility. If you are careful, you can even migrate those
containers. Running MAAS in an unprivileged container should be perfectly
possible https://github.com/CanonicalLtd/maas-docs/issues/700 - I am not
sure that the instructions that require a privileged container with loop
devices passed to it are relevant anymore.
An alternative is to use the lxd provider (which can connect to a remote
daemon, not only localhost) but this is only one daemon per provider. For
HA purposes you would need several LXDs on different hosts and for this
provider to support network spaces because you may have MAAS hosts located
in different layer 2 networks with different subnets used. Cross-model
relations could be used to have a model per LXD provider but I am not sure
this is the best approach - units would be on different models with no
shared unit-level leadership.
https://github.com/juju/juju/tree/develop/provider/lxd
With the new LXD clustering work it might be possible overcome this
limitation as well. I would assume LXD clustering to work on a per-rack
basis due to latency constraints while with MAAS in a data center you would
surely place different region controllers and rack controllers on different
racks (availability zones).
https://insights.ubuntu.com/2017/10/23/lxd-weekly-status-20-
authentication-conferences-more/
"Distributed database for LXD clustering"
If, by the time of LXD clustering release, there was support for
availability zones it would have solved the problem with a single control
plane for a Juju provider in the absence of MAAS.
An alternative to the above is just usage of bootstrap automation to set
up MAAS and then usage of Juju with charms for the rest of what you need.
Best Regards,
Dmitrii Shcherbakov
Field Software Engineer
IRC (freenode): Dmitrii-Sh

Post by James Beedy
Hello all,
I've got an HA maas setup at the datacenter, I had some trouble getting
the full HA bits super solid in the past, and thought it appropriate to try
charming up the new 2.3 MAAS snaps to see if I could make my life a bit
easier going forward.
I just took a quick first swipe at this, so please excuse the lack of
any tests.
I'm hoping I can kill 2 birds with 1 stone here by a) possibly getting
landing soon in reactive (and disseminate that feedback so others will be
nod if the steps I've taken here look to be moving in the right direction?
# Interface and layer
interface-maas: https://github.com/jamesbeedy/interface-maas
layer-maas: https://github.com/jamesbeedy/layer-maas
# MAAS charm
charmstore: https://jujucharms.com/u/jamesbeedy/maas/8
# Sample bundle
sample bundle: http://paste.ubuntu.com/26046016/ - (only for reference,
this won't actually deploy)
# juju status
`juju status`: http://paste.ubuntu.com/26045880/
Thanks,
James
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju

--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju

James Beedy

2017-11-27 13:53:55 UTC

Permalink

| I can also see that it took a few attempts to get there (the last machine
is #40 :)

It was a trying process to say the least - possibly I am one of few users
who would stick around to see it through..... which is actually great
because it creates a market for people to provide the service of providing
MAAS!

At least, with the MAAS charm, you can 1) create and add your vms, 2) juju
deploy bundle, 3) profit.

Instead of deploy #40 which is probably #100 by now + tears + 10 trips to
the datacenter + <add unpleasantry here>

Post by John Meinel
It does seem like Juju operating the LXD provider, but spanning multiple
machines would be an answer. I do believe that LXD itself is introducing
clustering into their API in the 18.04 cycle. Which would probably need
some updates on the Juju side to handle their targeted provisioning (create
a container for maas-postgresql on the 3rd machine in the LXD cluster).
But that might get you out of manual provisioning of a bunch of machines
and VMs to target everything.
We're certainly not at a point where you could just-do-that today.
I can also see that it took a few attempts to get there (the last machine
is #40 :)
John
=:->

Hi James,
This is an interesting approach, thanks for taking a shot at solving
this problem!
I thought of doing something similar a few months ago. The problematic
aspect here is the assumption of having a provider/substrate already
present for MAAS to be deployed - this is the chicken or the egg type of
problem.
If you would like to take the MAAS charm route, manual provider could be
used with Juju to do that with pre-created hosts (which may be
containers/VMs/hosts all in a single model with this provider, regardless
of how they were deployed). There would be hosts for a Juju controller(s)
and MAAS region/rack controllers in the end.
If you put both Juju controller and MAAS into containers, it gives you
some flexibility. If you are careful, you can even migrate those
containers. Running MAAS in an unprivileged container should be perfectly
possible https://github.com/CanonicalLtd/maas-docs/issues/700 - I am
not sure that the instructions that require a privileged container with
loop devices passed to it are relevant anymore.
An alternative is to use the lxd provider (which can connect to a remote
daemon, not only localhost) but this is only one daemon per provider. For
HA purposes you would need several LXDs on different hosts and for this
provider to support network spaces because you may have MAAS hosts located
in different layer 2 networks with different subnets used. Cross-model
relations could be used to have a model per LXD provider but I am not sure
this is the best approach - units would be on different models with no
shared unit-level leadership.
https://github.com/juju/juju/tree/develop/provider/lxd
With the new LXD clustering work it might be possible overcome this
limitation as well. I would assume LXD clustering to work on a per-rack
basis due to latency constraints while with MAAS in a data center you would
surely place different region controllers and rack controllers on different
racks (availability zones).
https://insights.ubuntu.com/2017/10/23/lxd-weekly-status-20-
authentication-conferences-more/
"Distributed database for LXD clustering"
If, by the time of LXD clustering release, there was support for
availability zones it would have solved the problem with a single control
plane for a Juju provider in the absence of MAAS.
An alternative to the above is just usage of bootstrap automation to set
up MAAS and then usage of Juju with charms for the rest of what you need.
Best Regards,
Dmitrii Shcherbakov
Field Software Engineer
IRC (freenode): Dmitrii-Sh

Post by James Beedy
Hello all,
I've got an HA maas setup at the datacenter, I had some trouble getting
the full HA bits super solid in the past, and thought it appropriate to try
charming up the new 2.3 MAAS snaps to see if I could make my life a bit
easier going forward.
I just took a quick first swipe at this, so please excuse the lack of
any tests.
I'm hoping I can kill 2 birds with 1 stone here by a) possibly getting
landing soon in reactive (and disseminate that feedback so others will be
nod if the steps I've taken here look to be moving in the right direction?
# Interface and layer
interface-maas: https://github.com/jamesbeedy/interface-maas
layer-maas: https://github.com/jamesbeedy/layer-maas
# MAAS charm
charmstore: https://jujucharms.com/u/jamesbeedy/maas/8
# Sample bundle
sample bundle: http://paste.ubuntu.com/26046016/ - (only for
reference, this won't actually deploy)
# juju status
`juju status`: http://paste.ubuntu.com/26045880/
Thanks,
James
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju

--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju

James Beedy

2017-11-27 15:20:13 UTC

Permalink

Oooooo .... possibly I'm on to something (last item/very bottom)
https://imgur.com/a/6Wyd0

Post by James Beedy
| I can also see that it took a few attempts to get there (the last
machine is #40 :)
It was a trying process to say the least - possibly I am one of few users
who would stick around to see it through..... which is actually great
because it creates a market for people to provide the service of providing
MAAS!
At least, with the MAAS charm, you can 1) create and add your vms, 2) juju
deploy bundle, 3) profit.
Instead of deploy #40 which is probably #100 by now + tears + 10 trips to
the datacenter + <add unpleasantry here>

Post by James Beedy
Dmitrii,
Thanks for the response.
When taking into account the overhead to get MAAS deployed as charms, it
definitely makes the LXD provider method you have described seem very
appealing. Some issues I've experienced trying to get MAAS HA deployed are
such that it seems like just a ton of infrastructure is needed to get MAAS
HA standing using the manual provider deploying the MAAS charms. You have
to provision/maintain the manual Juju controller cluster underneath MAAS,
just to have MAAS .... ugh
I found not only the sheer quantity of servers needed to get this
working quite unnerving, but also the manual ops I had to undergo to get it
all standing #snowflakes #custom.
I iterated on this a few times to see if I could come up with something
more manageable, and this is where I landed (brace yourself) [0]
What's going on there?
I created a model in JAAS, manually added 3 physical hosts across
different racks, then bootstrapped 4 virtual machines on each physical host
(1 vm for each postgresql, maas-region, maas-rack [1], juju-controller
(juju controllers for maas provider, to be checked into maas)) on each host.
I then also added my vms to my JAAS model so I could deploy the charms
to them (just postgresql and ubuntu at the time - the MAAS stuff got
manually installed and configured after the machiens had ubuntu deployed to
them in the model).
(ASIDE: I'm seeing I could probably do this one better by combining my
region and rack controller to the same vm, and colocating the postgresql
charm with the region+rack on the same vm, giving me ha of all components
using single vm on each host.)
I know there are probably a plethora of issues with what I've done here,
but it solved a few primary issues that seemed to outweigh the misuse.
1) Too many machines needed to get MAAS HA
2) Chicken or egg?
3) Snowflakes!!!
4) No clear path to support MAAS HA
My idea behind this was such that by using JAAS I am solving the chicken
or the egg issue, and reducing the machines/infra needed to get MAAS HA.
Deploying the MAAS infra with Juju eliminates the snowflake/lack of
tracking and chips at the "No clear path to support MAAS HA".
Thanks,
James
[0] http://paste.ubuntu.com/25891429/
[1] http://paste.ubuntu.com/26058033/
On Mon, Nov 27, 2017 at 12:09 AM, Dmitrii Shcherbakov <

Hi James,
This is an interesting approach, thanks for taking a shot at solving
this problem!
I thought of doing something similar a few months ago. The problematic
aspect here is the assumption of having a provider/substrate already
present for MAAS to be deployed - this is the chicken or the egg type of
problem.
If you would like to take the MAAS charm route, manual provider could
be used with Juju to do that with pre-created hosts (which may be
containers/VMs/hosts all in a single model with this provider, regardless
of how they were deployed). There would be hosts for a Juju controller(s)
and MAAS region/rack controllers in the end.
If you put both Juju controller and MAAS into containers, it gives you
some flexibility. If you are careful, you can even migrate those
containers. Running MAAS in an unprivileged container should be perfectly
possible https://github.com/CanonicalLtd/maas-docs/issues/700 - I am
not sure that the instructions that require a privileged container with
loop devices passed to it are relevant anymore.
An alternative is to use the lxd provider (which can connect to a
remote daemon, not only localhost) but this is only one daemon per
provider. For HA purposes you would need several LXDs on different hosts
and for this provider to support network spaces because you may have MAAS
hosts located in different layer 2 networks with different subnets used.
Cross-model relations could be used to have a model per LXD provider but I
am not sure this is the best approach - units would be on different models
with no shared unit-level leadership.
https://github.com/juju/juju/tree/develop/provider/lxd
With the new LXD clustering work it might be possible overcome this
limitation as well. I would assume LXD clustering to work on a per-rack
basis due to latency constraints while with MAAS in a data center you would
surely place different region controllers and rack controllers on different
racks (availability zones).
https://insights.ubuntu.com/2017/10/23/lxd-weekly-status-20-
authentication-conferences-more/
"Distributed database for LXD clustering"
If, by the time of LXD clustering release, there was support for
availability zones it would have solved the problem with a single control
plane for a Juju provider in the absence of MAAS.
An alternative to the above is just usage of bootstrap automation to
set up MAAS and then usage of Juju with charms for the rest of what you
need.
Best Regards,
Dmitrii Shcherbakov
Field Software Engineer
IRC (freenode): Dmitrii-Sh

Post by James Beedy
Hello all,
I've got an HA maas setup at the datacenter, I had some trouble
getting the full HA bits super solid in the past, and thought it
appropriate to try charming up the new 2.3 MAAS snaps to see if I could
make my life a bit easier going forward.
I just took a quick first swipe at this, so please excuse the lack of
any tests.
I'm hoping I can kill 2 birds with 1 stone here by a) possibly getting
landing soon in reactive (and disseminate that feedback so others will be
nod if the steps I've taken here look to be moving in the right direction?
# Interface and layer
interface-maas: https://github.com/jamesbeedy/interface-maas
layer-maas: https://github.com/jamesbeedy/layer-maas
# MAAS charm
charmstore: https://jujucharms.com/u/jamesbeedy/maas/8
# Sample bundle
sample bundle: http://paste.ubuntu.com/26046016/ - (only for
reference, this won't actually deploy)
# juju status
`juju status`: http://paste.ubuntu.com/26045880/
Thanks,
James
--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju

--
Juju mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
an/listinfo/juju