profile
viewpoint
Jeremy Gustie jgustie Williston, VT

carbonrelay/kustomize 0

Customization of kubernetes YAML configurations

jgustie/ghaction-docker-meta 0

:octocat: GitHub Action to extract metadata (tags, labels) for Docker

push eventthestormforge/optimize-controller

Brad Beam

commit sha 7c6a8f3dcd34aad7e4bf4552d3d1666109ea97a8

feat: Add support for patching Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 3a5b4f35e45dbee340ed54e842b0b267991b5e80

fix: fix up interaction with stdin Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha c6ca37784babf17dea2e6bc85a8b1ae53ccdb267

refactor: migrate to kio.pipelines This allows us to handle reading from stdin+files better. Otherwise we run into some wonky logic with trying to preserve the contents of o.In for multiple reads. Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 89d6536f01179c852ec00e78cf0e2ca98dff510c

fix: Correct usage descriptions Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 772d93f90ec16b6f415ca19738cbe955d259a88d

feat: Begin adding in application handling Still a WIP, but this starts to bring in support for handling using an application as the source. Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 506cb538532948e118c673dca8a7cd4081697746

feat: Add tests for patch application Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha fff85bde968e420070e4e15bdd8e2281f4bf5435

refactor: Switch to using filesys.Filesystem as backing store This changes the internal store to use filesys.Filesystem because it is a common base for both application generation and patching. Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 15a67a4ffbe7b968547510e55b66d5d197dee114

fix: Add default ContainerResources to application Missed adding DefaultContainerResourcesSelectors to the application which meant we would not generate any patches for the experiment. Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 2182e6be7b6c069544db03d9c4772e65db48ae21

chore: Remove unused functions Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 1275238381de57c0d7902b5e0c66438c5517dc55

feat: Fix up handling of non application input Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 92a1762873829cfb69c2020554bf1a292c959120

refactor: Reimplement kio pipelines This brings kio pipelines back into the picture so we can more easily manipulate and identify resources from multidoc yaml inputs. Signed-off-by: Brad Beam <brad.beam@stormforge.io>

view details

Brad Beam

commit sha 7e1ea9ced31d27147cc478656d051341f7da318e

chore: Rename command from patch to export Signed-off-by: Brad Beam <brad.beam@stormforge.io>

view details

Brad Beam

commit sha 5132d87002e76ca26c8c85b5dd1fa8abaeb1c292

refactor: Make trial name a positional arg Signed-off-by: Brad Beam <brad.beam@stormforge.io>

view details

Brad Beam

commit sha ae3e6527910e6a337d88262f695049feefebc584

chore: Rename import paths Signed-off-by: Brad Beam <brad.beam@b-rad.info>

view details

Brad Beam

commit sha 6e3630448f87e236b431df8b4c430ee5e2754258

refactor: Cleanup from review Signed-off-by: Brad Beam <brad.beam@b-rad.info>

view details

Brad Beam

commit sha 322a2d06349a47130eef75765951df3d901ec5b2

refactor: Make export level up There were a number of previous use cases I hadnt considered. This should address them. - Handle filtering application to only relevant trial/experiment - Handle application.Resources as kustomize resources ( not only flat files ) - Fix/add support for jsonpatches - Allow kustomize patches to be inline patches Signed-off-by: Brad Beam <brad.beam@b-rad.info>

view details

Brad Beam

commit sha ae490ebccb040aa710af9c3d53abee8f337a27d1

feat: Add support for patched-target and patch flags Signed-off-by: Brad Beam <brad.beam@stormforge.io>

view details

Brad Beam

commit sha 35719584fa7c766da80f765efa2d17763eb4271a

fix: Correct usage message, add filename requirements Signed-off-by: Brad Beam <brad.beam@stormforge.io>

view details

Brad Beam

commit sha 6b513fb792d5f1517f5c39e942e3c1fcd1091fd5

Merge pull request #242 from bradbeam/patchcommand Add support for export command

view details

push time in 4 days

PR merged thestormforge/optimize-controller

Add support for export command

Looks like #138 got closed with the generate merge and I can't seem to reopen it.

This introduces an export command to generate patched resources. This can be run in 3 different ways.

  1. redskyctl export --filename app.yaml test-experiment-from-file-250 This will produce all the necessary resources to deploy the application using the parameters from trial 250. ( I've left the output out since its a considerable amount of output.

  2. redskyctl export --filename app.yaml test-experiment-from-file-250 --patch This produces the patch(es) that can be applied directly to a Kubernetes cluster.

[20:35:40] (0):~/go/src/github.com/thestormforge/optimize-controller
% $controller/funsies/redskyctl export --filename app.yaml test-experiment-from-file-250 --patch     
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"voting-service","namespace":"voting-webapp"},"spec":{"replicas":1,"template":{"spec":{"containers":[{"name":"voting-service","resources":{"limits":{"cpu":"100m","memory":"128M"},"requests":{"cpu":"100m","memory":"128M"}}}]}}}}
  1. redskyctl export --filename app.yaml test-experiment-from-file-250 --patched-target This produces the manifests for only the patched resources.
[20:35:14] (0):~/go/src/github.com/thestormforge/optimize-controller
% $controller/funsies/redskyctl-bin export --filename app.yaml test-experiment-from-file-250 --patched-target
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: voting-app
    component: voting-service
  name: voting-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-app
      component: voting-service
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: voting-app
        component: voting-service
      name: voting-service
    spec:
      containers:
      - image: dockersamples/examplevotingapp_vote
        name: voting-service
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          limits:
            cpu: 100m
            memory: 128M
          requests:
            cpu: 100m
            memory: 128M
+1220 -6

0 comment

7 changed files

bradbeam

pr closed time in 4 days

PR opened thestormforge/optimize-controller

Add support for export command

Looks like #138 got closed with the generate merge and I can't seem to reopen it.

This introduces an export command to generate patched resources. This can be run in 3 different ways.

  1. redskyctl export --filename app.yaml test-experiment-from-file-250 This will produce all the necessary resources to deploy the application using the parameters from trial 250. ( I've left the output out since its a considerable amount of output.

  2. redskyctl export --filename app.yaml test-experiment-from-file-250 --patch This produces the patch(es) that can be applied directly to a Kubernetes cluster.

[20:35:40] (0):~/go/src/github.com/thestormforge/optimize-controller
% $controller/funsies/redskyctl export --filename app.yaml test-experiment-from-file-250 --patch     
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"voting-service","namespace":"voting-webapp"},"spec":{"replicas":1,"template":{"spec":{"containers":[{"name":"voting-service","resources":{"limits":{"cpu":"100m","memory":"128M"},"requests":{"cpu":"100m","memory":"128M"}}}]}}}}
  1. redskyctl export --filename app.yaml test-experiment-from-file-250 --patched-target This produces the manifests for only the patched resources.
[20:35:14] (0):~/go/src/github.com/thestormforge/optimize-controller
% $controller/funsies/redskyctl-bin export --filename app.yaml test-experiment-from-file-250 --patched-target
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: voting-app
    component: voting-service
  name: voting-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-app
      component: voting-service
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: voting-app
        component: voting-service
      name: voting-service
    spec:
      containers:
      - image: dockersamples/examplevotingapp_vote
        name: voting-service
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          limits:
            cpu: 100m
            memory: 128M
          requests:
            cpu: 100m
            memory: 128M
+1216 -5

0 comment

6 changed files

pr created time in 5 days

delete branch thestormforge/optimize-controller

delete branch : add/generate-experiment

delete time in 5 days

create barnchthestormforge/optimize-controller

branch : add/generate-experiment

created branch time in 5 days

Pull request review commentthestormforge/optimize-controller

Deprecate results

 limitations under the License. package results  import (-	"context" 	"fmt"-	"net/http"-	"net/http/httputil" 	"os/user"-	"strings" 	"time"  	"github.com/pkg/browser" 	"github.com/redskyops/redskyops-controller/redskyctl/internal/commander"-	"github.com/redskyops/redskyops-ui/v2/ui" 	"github.com/spf13/cobra"-	"github.com/thestormforge/optimize-go/pkg/redskyapi"-	"golang.org/x/oauth2" )  // Options is the configuration for displaying the results UI type Options struct {-	// Config is the Red Sky Configuration to proxy-	Config redskyapi.Config 	// IOStreams are used to access the standard process streams 	commander.IOStreams -	// ServerAddress is the address to listen on (defaults to an ephemeral port) 	ServerAddress string-	// DisplayURL just prints the URL instead of opening the default browser-	DisplayURL bool-	// IdleTimeout is the time between heartbeats to the "/health" endpoint required to keep the server up (defaults to 5 seconds)-	IdleTimeout time.Duration+	DisplayURL    bool+	IdleTimeout   time.Duration }  // NewCommand creates a new command for displaying the results UI func NewCommand(o *Options) *cobra.Command { 	cmd := &cobra.Command{-		Use:   "results",-		Short: "Serve a visualization of the results",--		PreRun: func(cmd *cobra.Command, args []string) {-			commander.SetStreams(&o.IOStreams, cmd)-			o.Complete()-		},-		RunE: commander.WithContextE(o.results),+		Use:        "results",+		Short:      "View a visualization of the results",+		Deprecated: "you can now access your results anytime using the web interface",++		PreRun: commander.StreamsPreRun(&o.IOStreams),+		RunE:   commander.WithoutArgsE(o.results), 	} -	cmd.Flags().StringVar(&o.ServerAddress, "address", "", "address to listen on")+	// Keep the flags so we don't fail, but mark the all as hidden+	cmd.Flags().StringVar(&o.ServerAddress, "address", "", "ignored for compatibility") 	cmd.Flags().BoolVar(&o.DisplayURL, "url", false, "display the URL instead of opening a browser")-	cmd.Flags().DurationVar(&o.IdleTimeout, "idle-timeout", 5*time.Second, "set the heartbeat interval (0 to ignore heartbeats)")+	cmd.Flags().DurationVar(&o.IdleTimeout, "idle-timeout", 5*time.Second, "ignored for compatibility")+	_ = cmd.Flags().MarkHidden("address")+	_ = cmd.Flags().MarkHidden("url") 	_ = cmd.Flags().MarkHidden("idle-timeout")  	return cmd } -func (o *Options) Complete() {-	// Default to listening on an ephemeral port-	if o.ServerAddress == "" {-		o.ServerAddress = ":0"-	}--	// If we are just printing a URL we can't rely on the heartbeat to keep the process alive-	if o.DisplayURL {-		o.IdleTimeout = 0-	}-}--func (o *Options) results(ctx context.Context) error {-	// Create the router to match requests-	router := http.NewServeMux()-	if err := o.handleAPI(ctx, router, "/v1/"); err != nil {-		return err-	}-	o.handleUI(router, "/ui/")-	o.handleLiveness(router, "/health")--	// Create the server-	server := commander.NewContextServer(ctx, router,-		commander.WithServerOptions(o.configureServer),-		commander.ShutdownOnInterrupt(func() { _, _ = fmt.Fprintln(o.Out) }),-		commander.ShutdownOnIdle(o.IdleTimeout, func() { _, _ = fmt.Fprintln(o.Out) }),-		commander.HandleStart(o.openBrowser))--	// Start the server, this will block until someone calls 'shutdown' from above-	return server.ListenAndServe()-}--func (o *Options) configureServer(srv *http.Server) {-	srv.Addr = o.ServerAddress-	srv.ReadTimeout = 5 * time.Second-	srv.WriteTimeout = 10 * time.Second-	srv.IdleTimeout = 15 * time.Second-}--func (o *Options) openBrowser(loc string) error {+func (o *Options) results() error { 	u, err := user.Current() 	if err != nil { 		return err 	} +	loc := "https://app.stormforge.io/experiments"

Should we/can we pull this in from the config? More of a dev issue

jgustie

comment created time in 9 days

push eventthestormforge/optimize-controller

Brad Beam

commit sha dfb684718d9632814624c243297c624e0ea53261

fix: Patch trial job instead of delete it We can suspend/interrupt a pending trial job by setting the parallelism to 0. This should allow us to preserve trial run history and handle cases where the trial job fails to schedule because of resource constraints. Signed-off-by: Brad Beam <brad.beam@b-rad.info>

view details

Brad Beam

commit sha 77b8b51e0f7254e4764510fa3acb13964188a2e2

chore: Clean up log statement Signed-off-by: Brad Beam <brad.beam@b-rad.info>

view details

Brad Beam

commit sha 19ec84afa13f39d36491a50445cc9c8f500132dd

Merge pull request #238 from bradbeam/jobs fix: Patch trial job instead of delete it

view details

push time in 9 days

PR merged thestormforge/optimize-controller

fix: Patch trial job instead of delete it

We can suspend/interrupt a pending trial job by setting the parallelism to 0. This should allow us to preserve trial run history and handle cases where the trial job fails to schedule because of resource constraints.

Signed-off-by: Brad Beam brad.beam@b-rad.info

+28 -16

0 comment

2 changed files

bradbeam

pr closed time in 9 days

Pull request review commentthestormforge/optimize-controller

fix: Patch trial job instead of delete it

 func (r *TrialJobReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { 	}  	// Create a new job if necessary-	if len(jobList.Items) == 0 {-		// Insert a "sleep" between "ready" and the trial job-		if ids := time.Duration(t.Spec.InitialDelaySeconds) * time.Second; ids > 0 {-			for _, c := range t.Status.Conditions {-				if c.Type == redskyv1beta1.TrialReady {-					startTime := c.LastTransitionTime.Add(ids)-					if startTime.After(now.Time) {-						return ctrl.Result{RequeueAfter: startTime.Sub(now.Time)}, nil-					}+	if len(jobList.Items) > 0 {+		return ctrl.Result{}, nil+	}++	// Insert a "sleep" between "ready" and the trial job+	if ids := time.Duration(t.Spec.InitialDelaySeconds) * time.Second; ids > 0 {+		for _, c := range t.Status.Conditions {+			if c.Type == redskyv1beta1.TrialReady {+				startTime := c.LastTransitionTime.Add(ids)+				if startTime.After(now.Time) {+					return ctrl.Result{RequeueAfter: startTime.Sub(now.Time)}, nil

This was more of a cosmetic change and can be reverted if needed. Wanted to unwrap the code block by returning early.

bradbeam

comment created time in 9 days

PR opened thestormforge/optimize-controller

fix: Patch trial job instead of delete it

We can suspend/interrupt a pending trial job by setting the parallelism to 0. This should allow us to preserve trial run history and handle cases where the trial job fails to schedule because of resource constraints.

Signed-off-by: Brad Beam brad.beam@b-rad.info

+28 -16

0 comment

2 changed files

pr created time in 9 days

Pull request review commentthestormforge/optimize-controller

Add support for patch command

+/*+Copyright 2020 GramLabs, Inc.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package export++import (+	"bytes"+	"context"+	"encoding/json"+	"fmt"+	"io/ioutil"+	"path/filepath"+	"strconv"+	"strings"++	app "github.com/redskyops/redskyops-controller/api/apps/v1alpha1"+	redsky "github.com/redskyops/redskyops-controller/api/v1beta1"+	"github.com/redskyops/redskyops-controller/internal/experiment"+	"github.com/redskyops/redskyops-controller/internal/patch"+	"github.com/redskyops/redskyops-controller/internal/server"+	"github.com/redskyops/redskyops-controller/internal/template"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/commander"+	experimentctl "github.com/redskyops/redskyops-controller/redskyctl/internal/commands/generate/experiment"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/kustomize"+	"github.com/redskyops/redskyops-go/pkg/config"+	experimentsapi "github.com/redskyops/redskyops-go/pkg/redskyapi/experiments/v1alpha1"+	"github.com/spf13/cobra"+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"+	"k8s.io/apimachinery/pkg/labels"+	"sigs.k8s.io/kustomize/api/filesys"+	"sigs.k8s.io/kustomize/api/resid"+	"sigs.k8s.io/kustomize/api/types"+	"sigs.k8s.io/kustomize/kyaml/kio"+	"sigs.k8s.io/kustomize/kyaml/yaml"+)++// Options are the configuration options for creating a patched experiment+type Options struct {+	// Config is the Red Sky Configuration used to generate the controller installation+	Config *config.RedSkyConfig+	// ExperimentsAPI is used to interact with the Red Sky Experiments API+	ExperimentsAPI experimentsapi.API+	// IOStreams are used to access the standard process streams+	commander.IOStreams++	inputFiles []string+	trialName  string++	// This is used for testing+	Fs          filesys.FileSystem+	inputData   []byte+	experiment  *redsky.Experiment+	application *app.Application+	resources   map[string]struct{}+}++// NewCommand creates a command for performing an export+func NewCommand(o *Options) *cobra.Command {+	cmd := &cobra.Command{+		Use:   "export",+		Short: "Export trial parameters to an application or experiment",+		Long:  "Export trial parameters to an application or experiment from the specified trial",++		PreRunE: func(cmd *cobra.Command, args []string) error {+			commander.SetStreams(&o.IOStreams, cmd)++			var err error+			if o.ExperimentsAPI == nil {+				err = commander.SetExperimentsAPI(&o.ExperimentsAPI, o.Config, cmd)+			}++			if len(args) != 1 {+				return fmt.Errorf("a trial name must be specified")+			}++			o.trialName = args[0]++			return err+		},+		RunE: commander.WithContextE(o.runner),+	}++	cmd.Flags().StringSliceVar(&o.inputFiles, "file", []string{""}, "experiment and related manifests to export, - for stdin")++	return cmd+}++func (o *Options) readInput() error {+	// Do an in memory filesystem so we can properly handle stdin+	if o.Fs == nil {+		o.Fs = filesys.MakeFsInMemory()+	}++	if o.resources == nil {+		o.resources = make(map[string]struct{})+	}++	kioInputs := []kio.Reader{}++	for _, filename := range o.inputFiles {+		r, err := o.IOStreams.OpenFile(filename)+		if err != nil {+			return err+		}+		defer r.Close()++		data, err := ioutil.ReadAll(r)+		if err != nil {+			return err+		}++		if filename == "-" {+			filename = "stdin.yaml"+		}++		if err := o.Fs.WriteFile(filepath.Base(filename), data); err != nil {+			return err+		}++		kioInputs = append(kioInputs, &kio.ByteReader{Reader: bytes.NewReader(data)})++		// Track all of the input files so we can use them as kustomize resources later on+		o.resources[filepath.Base(filename)] = struct{}{}+	}++	var inputsBuf bytes.Buffer++	// Aggregate all inputs+	allInput := kio.Pipeline{+		Inputs:  kioInputs,+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &inputsBuf}},+	}+	if err := allInput.Execute(); err != nil {+		return err+	}++	o.inputData = inputsBuf.Bytes()++	return nil+}++func (o *Options) extractApplication() error {+	var appBuf bytes.Buffer++	// Render Experiment+	appInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Application"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &appBuf}},+	}+	if err := appInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an application since we'll handle this later+	if appBuf.Len() == 0 {+		return nil+	}++	o.application = &app.Application{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&appBuf), o.application)+}++func (o *Options) extractExperiment() error {+	var experimentBuf bytes.Buffer++	// Render Experiment+	experimentInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Experiment"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &experimentBuf}},+	}+	if err := experimentInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an experiment since we'll handle this later+	if experimentBuf.Len() == 0 {+		return nil+	}++	o.experiment = &redsky.Experiment{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&experimentBuf), o.experiment)+}++// filter returns a filter function to exctract a specified `kind` from the input.+func filter(kind string) kio.FilterFunc {+	return func(input []*yaml.RNode) ([]*yaml.RNode, error) {+		var output kio.ResourceNodeSlice+		for i := range input {+			m, err := input[i].GetMeta()+			if err != nil {+				return nil, err+			}+			if m.Kind != kind {+				continue+			}+			output = append(output, input[i])+		}+		return output, nil+	}+}++func (o *Options) runner(ctx context.Context) error {+	if o.trialName == "" {+		return fmt.Errorf("a trial name must be specified")+	}++	if err := o.readInput(); err != nil {+		return err+	}++	// See if we have been given an applcation+	if err := o.extractApplication(); err != nil {+		return fmt.Errorf("got an error when looking for application: %w", err)+	}++	// See if we have been given an experiment+	if err := o.extractExperiment(); err != nil {+		return fmt.Errorf("got an error when looking for experiment: %w", err)+	}++	switch {+	case o.application != nil:+		// Reset/Restrict application resources to only those specified by the application+		// and resources generated by the generator+		o.resources = make(map[string]struct{})+		for _, file := range o.application.Resources {+			o.resources[file] = struct{}{}+		}++		if err := o.generateExperiment(); err != nil {+			return err+		}+	case o.experiment != nil:+		// Dont need to do anything special+	default:+		return fmt.Errorf("unable to identify an experiment or application")+	}++	// At this point we must have an experiment+	if o.experiment == nil {+		return fmt.Errorf("unable to find an experiment")+	}++	// look up trial from api+	trialItem, err := o.getTrialByID(ctx, o.experiment.Name)+	if err != nil {+		return err+	}++	trial := &redsky.Trial{}+	experiment.PopulateTrialFromTemplate(o.experiment, trial)+	server.ToClusterTrial(trial, &trialItem.TrialAssignments)++	// render patches+	var patches map[string]types.Patch+	patches, err = createKustomizePatches(o.experiment.Spec.Patches, trial)+	if err != nil {+		return err+	}++	resourceNames := make([]string, 0, len(o.resources))+	for name := range o.resources {+		resourceNames = append(resourceNames, name)++	}++	yamls, err := kustomize.Yamls(+		kustomize.WithFS(o.Fs),+		kustomize.WithResourceNames(resourceNames),+		kustomize.WithPatches(patches),+	)+	if err != nil {+		return err+	}++	fmt.Fprintln(o.Out, string(yamls))++	return nil+}++func (o *Options) generateExperiment() error {+	gen := experimentctl.NewGenerator(o.Fs)+	gen.Application = *o.application+	gen.ContainerResourcesSelectors = experimentctl.DefaultContainerResourcesSelectors()++	if gen.Application.Parameters != nil && gen.Application.Parameters.ContainerResources != nil {+		ls := labels.Set(gen.Application.Parameters.ContainerResources.Labels).String()++		for i := range gen.ContainerResourcesSelectors {+			gen.ContainerResourcesSelectors[i].LabelSelector = ls+		}+	}+

this should go away

bradbeam

comment created time in 10 days

Pull request review commentthestormforge/optimize-controller

Add support for patch command

+/*+Copyright 2020 GramLabs, Inc.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package export++import (+	"bytes"+	"context"+	"encoding/json"+	"fmt"+	"io/ioutil"+	"path/filepath"+	"strconv"+	"strings"++	app "github.com/redskyops/redskyops-controller/api/apps/v1alpha1"+	redsky "github.com/redskyops/redskyops-controller/api/v1beta1"+	"github.com/redskyops/redskyops-controller/internal/experiment"+	"github.com/redskyops/redskyops-controller/internal/patch"+	"github.com/redskyops/redskyops-controller/internal/server"+	"github.com/redskyops/redskyops-controller/internal/template"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/commander"+	experimentctl "github.com/redskyops/redskyops-controller/redskyctl/internal/commands/generate/experiment"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/kustomize"+	"github.com/redskyops/redskyops-go/pkg/config"+	experimentsapi "github.com/redskyops/redskyops-go/pkg/redskyapi/experiments/v1alpha1"+	"github.com/spf13/cobra"+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"+	"k8s.io/apimachinery/pkg/labels"+	"sigs.k8s.io/kustomize/api/filesys"+	"sigs.k8s.io/kustomize/api/resid"+	"sigs.k8s.io/kustomize/api/types"+	"sigs.k8s.io/kustomize/kyaml/kio"+	"sigs.k8s.io/kustomize/kyaml/yaml"+)++// Options are the configuration options for creating a patched experiment+type Options struct {+	// Config is the Red Sky Configuration used to generate the controller installation+	Config *config.RedSkyConfig+	// ExperimentsAPI is used to interact with the Red Sky Experiments API+	ExperimentsAPI experimentsapi.API+	// IOStreams are used to access the standard process streams+	commander.IOStreams++	inputFiles []string+	trialName  string++	// This is used for testing+	Fs          filesys.FileSystem+	inputData   []byte+	experiment  *redsky.Experiment+	application *app.Application+	resources   map[string]struct{}+}++// NewCommand creates a command for performing an export+func NewCommand(o *Options) *cobra.Command {+	cmd := &cobra.Command{+		Use:   "export",+		Short: "Export trial parameters to an application or experiment",+		Long:  "Export trial parameters to an application or experiment from the specified trial",++		PreRunE: func(cmd *cobra.Command, args []string) error {+			commander.SetStreams(&o.IOStreams, cmd)++			var err error+			if o.ExperimentsAPI == nil {+				err = commander.SetExperimentsAPI(&o.ExperimentsAPI, o.Config, cmd)+			}++			if len(args) != 1 {+				return fmt.Errorf("a trial name must be specified")+			}++			o.trialName = args[0]++			return err+		},+		RunE: commander.WithContextE(o.runner),+	}++	cmd.Flags().StringSliceVar(&o.inputFiles, "file", []string{""}, "experiment and related manifests to export, - for stdin")++	return cmd+}++func (o *Options) readInput() error {+	// Do an in memory filesystem so we can properly handle stdin+	if o.Fs == nil {+		o.Fs = filesys.MakeFsInMemory()+	}++	if o.resources == nil {+		o.resources = make(map[string]struct{})+	}++	kioInputs := []kio.Reader{}++	for _, filename := range o.inputFiles {+		r, err := o.IOStreams.OpenFile(filename)+		if err != nil {+			return err+		}+		defer r.Close()++		data, err := ioutil.ReadAll(r)+		if err != nil {+			return err+		}++		if filename == "-" {+			filename = "stdin.yaml"+		}++		if err := o.Fs.WriteFile(filepath.Base(filename), data); err != nil {+			return err+		}++		kioInputs = append(kioInputs, &kio.ByteReader{Reader: bytes.NewReader(data)})++		// Track all of the input files so we can use them as kustomize resources later on+		o.resources[filepath.Base(filename)] = struct{}{}+	}++	var inputsBuf bytes.Buffer++	// Aggregate all inputs+	allInput := kio.Pipeline{+		Inputs:  kioInputs,+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &inputsBuf}},+	}+	if err := allInput.Execute(); err != nil {+		return err+	}++	o.inputData = inputsBuf.Bytes()++	return nil+}++func (o *Options) extractApplication() error {+	var appBuf bytes.Buffer++	// Render Experiment+	appInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Application"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &appBuf}},+	}+	if err := appInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an application since we'll handle this later+	if appBuf.Len() == 0 {+		return nil+	}++	o.application = &app.Application{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&appBuf), o.application)+}++func (o *Options) extractExperiment() error {+	var experimentBuf bytes.Buffer++	// Render Experiment+	experimentInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Experiment"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &experimentBuf}},+	}+	if err := experimentInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an experiment since we'll handle this later+	if experimentBuf.Len() == 0 {+		return nil+	}++	o.experiment = &redsky.Experiment{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&experimentBuf), o.experiment)+}++// filter returns a filter function to exctract a specified `kind` from the input.+func filter(kind string) kio.FilterFunc {+	return func(input []*yaml.RNode) ([]*yaml.RNode, error) {+		var output kio.ResourceNodeSlice+		for i := range input {+			m, err := input[i].GetMeta()+			if err != nil {+				return nil, err+			}+			if m.Kind != kind {+				continue+			}+			output = append(output, input[i])+		}+		return output, nil+	}+}++func (o *Options) runner(ctx context.Context) error {+	if o.trialName == "" {+		return fmt.Errorf("a trial name must be specified")+	}++	if err := o.readInput(); err != nil {+		return err+	}++	// See if we have been given an applcation+	if err := o.extractApplication(); err != nil {+		return fmt.Errorf("got an error when looking for application: %w", err)+	}++	// See if we have been given an experiment+	if err := o.extractExperiment(); err != nil {+		return fmt.Errorf("got an error when looking for experiment: %w", err)+	}++	switch {+	case o.application != nil:+		// Reset/Restrict application resources to only those specified by the application+		// and resources generated by the generator+		o.resources = make(map[string]struct{})+		for _, file := range o.application.Resources {+			o.resources[file] = struct{}{}+		}++		if err := o.generateExperiment(); err != nil {+			return err+		}+	case o.experiment != nil:+		// Dont need to do anything special+	default:+		return fmt.Errorf("unable to identify an experiment or application")+	}++	// At this point we must have an experiment+	if o.experiment == nil {+		return fmt.Errorf("unable to find an experiment")+	}++	// look up trial from api+	trialItem, err := o.getTrialByID(ctx, o.experiment.Name)+	if err != nil {+		return err+	}++	trial := &redsky.Trial{}+	experiment.PopulateTrialFromTemplate(o.experiment, trial)+	server.ToClusterTrial(trial, &trialItem.TrialAssignments)++	// render patches+	var patches map[string]types.Patch+	patches, err = createKustomizePatches(o.experiment.Spec.Patches, trial)+	if err != nil {+		return err+	}++	resourceNames := make([]string, 0, len(o.resources))+	for name := range o.resources {+		resourceNames = append(resourceNames, name)++	}++	yamls, err := kustomize.Yamls(+		kustomize.WithFS(o.Fs),+		kustomize.WithResourceNames(resourceNames),+		kustomize.WithPatches(patches),+	)+	if err != nil {+		return err+	}++	fmt.Fprintln(o.Out, string(yamls))++	return nil+}++func (o *Options) generateExperiment() error {

add app.filterByExperiment(o.trialName(minus id)) to reduce the application down to only the stuff we care about

bradbeam

comment created time in 10 days

Pull request review commentthestormforge/optimize-controller

Add support for patch command

+/*+Copyright 2020 GramLabs, Inc.++Licensed under the Apache License, Version 2.0 (the "License");+you may not use this file except in compliance with the License.+You may obtain a copy of the License at++    http://www.apache.org/licenses/LICENSE-2.0++Unless required by applicable law or agreed to in writing, software+distributed under the License is distributed on an "AS IS" BASIS,+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+See the License for the specific language governing permissions and+limitations under the License.+*/++package export++import (+	"bytes"+	"context"+	"encoding/json"+	"fmt"+	"io/ioutil"+	"path/filepath"+	"strconv"+	"strings"++	app "github.com/redskyops/redskyops-controller/api/apps/v1alpha1"+	redsky "github.com/redskyops/redskyops-controller/api/v1beta1"+	"github.com/redskyops/redskyops-controller/internal/experiment"+	"github.com/redskyops/redskyops-controller/internal/patch"+	"github.com/redskyops/redskyops-controller/internal/server"+	"github.com/redskyops/redskyops-controller/internal/template"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/commander"+	experimentctl "github.com/redskyops/redskyops-controller/redskyctl/internal/commands/generate/experiment"+	"github.com/redskyops/redskyops-controller/redskyctl/internal/kustomize"+	"github.com/redskyops/redskyops-go/pkg/config"+	experimentsapi "github.com/redskyops/redskyops-go/pkg/redskyapi/experiments/v1alpha1"+	"github.com/spf13/cobra"+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"+	"k8s.io/apimachinery/pkg/labels"+	"sigs.k8s.io/kustomize/api/filesys"+	"sigs.k8s.io/kustomize/api/resid"+	"sigs.k8s.io/kustomize/api/types"+	"sigs.k8s.io/kustomize/kyaml/kio"+	"sigs.k8s.io/kustomize/kyaml/yaml"+)++// Options are the configuration options for creating a patched experiment+type Options struct {+	// Config is the Red Sky Configuration used to generate the controller installation+	Config *config.RedSkyConfig+	// ExperimentsAPI is used to interact with the Red Sky Experiments API+	ExperimentsAPI experimentsapi.API+	// IOStreams are used to access the standard process streams+	commander.IOStreams++	inputFiles []string+	trialName  string++	// This is used for testing+	Fs          filesys.FileSystem+	inputData   []byte+	experiment  *redsky.Experiment+	application *app.Application+	resources   map[string]struct{}+}++// NewCommand creates a command for performing an export+func NewCommand(o *Options) *cobra.Command {+	cmd := &cobra.Command{+		Use:   "export",+		Short: "Export trial parameters to an application or experiment",+		Long:  "Export trial parameters to an application or experiment from the specified trial",++		PreRunE: func(cmd *cobra.Command, args []string) error {+			commander.SetStreams(&o.IOStreams, cmd)++			var err error+			if o.ExperimentsAPI == nil {+				err = commander.SetExperimentsAPI(&o.ExperimentsAPI, o.Config, cmd)+			}++			if len(args) != 1 {+				return fmt.Errorf("a trial name must be specified")+			}++			o.trialName = args[0]++			return err+		},+		RunE: commander.WithContextE(o.runner),+	}++	cmd.Flags().StringSliceVar(&o.inputFiles, "file", []string{""}, "experiment and related manifests to export, - for stdin")++	return cmd+}++func (o *Options) readInput() error {+	// Do an in memory filesystem so we can properly handle stdin+	if o.Fs == nil {+		o.Fs = filesys.MakeFsInMemory()+	}++	if o.resources == nil {+		o.resources = make(map[string]struct{})+	}++	kioInputs := []kio.Reader{}++	for _, filename := range o.inputFiles {+		r, err := o.IOStreams.OpenFile(filename)+		if err != nil {+			return err+		}+		defer r.Close()++		data, err := ioutil.ReadAll(r)+		if err != nil {+			return err+		}++		if filename == "-" {+			filename = "stdin.yaml"+		}++		if err := o.Fs.WriteFile(filepath.Base(filename), data); err != nil {+			return err+		}++		kioInputs = append(kioInputs, &kio.ByteReader{Reader: bytes.NewReader(data)})++		// Track all of the input files so we can use them as kustomize resources later on+		o.resources[filepath.Base(filename)] = struct{}{}+	}++	var inputsBuf bytes.Buffer++	// Aggregate all inputs+	allInput := kio.Pipeline{+		Inputs:  kioInputs,+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &inputsBuf}},+	}+	if err := allInput.Execute(); err != nil {+		return err+	}++	o.inputData = inputsBuf.Bytes()++	return nil+}++func (o *Options) extractApplication() error {+	var appBuf bytes.Buffer++	// Render Experiment+	appInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Application"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &appBuf}},+	}+	if err := appInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an application since we'll handle this later+	if appBuf.Len() == 0 {+		return nil+	}++	o.application = &app.Application{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&appBuf), o.application)+}++func (o *Options) extractExperiment() error {+	var experimentBuf bytes.Buffer++	// Render Experiment+	experimentInput := kio.Pipeline{+		Inputs:  []kio.Reader{&kio.ByteReader{Reader: bytes.NewReader(o.inputData)}},+		Filters: []kio.Filter{kio.FilterFunc(filter("Experiment"))},+		Outputs: []kio.Writer{kio.ByteWriter{Writer: &experimentBuf}},+	}+	if err := experimentInput.Execute(); err != nil {+		return err+	}++	// We don't want to bail if we cant find an experiment since we'll handle this later+	if experimentBuf.Len() == 0 {+		return nil+	}++	o.experiment = &redsky.Experiment{}++	return commander.NewResourceReader().ReadInto(ioutil.NopCloser(&experimentBuf), o.experiment)+}++// filter returns a filter function to exctract a specified `kind` from the input.+func filter(kind string) kio.FilterFunc {+	return func(input []*yaml.RNode) ([]*yaml.RNode, error) {+		var output kio.ResourceNodeSlice+		for i := range input {+			m, err := input[i].GetMeta()+			if err != nil {+				return nil, err+			}+			if m.Kind != kind {+				continue+			}+			output = append(output, input[i])+		}+		return output, nil+	}+}++func (o *Options) runner(ctx context.Context) error {+	if o.trialName == "" {+		return fmt.Errorf("a trial name must be specified")+	}++	if err := o.readInput(); err != nil {+		return err+	}++	// See if we have been given an applcation+	if err := o.extractApplication(); err != nil {+		return fmt.Errorf("got an error when looking for application: %w", err)+	}++	// See if we have been given an experiment+	if err := o.extractExperiment(); err != nil {+		return fmt.Errorf("got an error when looking for experiment: %w", err)+	}++	switch {+	case o.application != nil:+		// Reset/Restrict application resources to only those specified by the application+		// and resources generated by the generator+		o.resources = make(map[string]struct{})+		for _, file := range o.application.Resources {+			o.resources[file] = struct{}{}+		}++		if err := o.generateExperiment(); err != nil {+			return err+		}+	case o.experiment != nil:+		// Dont need to do anything special+	default:+		return fmt.Errorf("unable to identify an experiment or application")+	}++	// At this point we must have an experiment+	if o.experiment == nil {+		return fmt.Errorf("unable to find an experiment")+	}++	// look up trial from api+	trialItem, err := o.getTrialByID(ctx, o.experiment.Name)+	if err != nil {+		return err+	}++	trial := &redsky.Trial{}+	experiment.PopulateTrialFromTemplate(o.experiment, trial)+	server.ToClusterTrial(trial, &trialItem.TrialAssignments)++	// render patches+	var patches map[string]types.Patch+	patches, err = createKustomizePatches(o.experiment.Spec.Patches, trial)+	if err != nil {+		return err+	}++	resourceNames := make([]string, 0, len(o.resources))+	for name := range o.resources {+		resourceNames = append(resourceNames, name)++	}++	yamls, err := kustomize.Yamls(+		kustomize.WithFS(o.Fs),+		kustomize.WithResourceNames(resourceNames),+		kustomize.WithPatches(patches),+	)+	if err != nil {+		return err+	}++	fmt.Fprintln(o.Out, string(yamls))++	return nil+}++func (o *Options) generateExperiment() error {+	gen := experimentctl.NewGenerator(o.Fs)+	gen.Application = *o.application+	gen.ContainerResourcesSelectors = experimentctl.DefaultContainerResourcesSelectors()++	if gen.Application.Parameters != nil && gen.Application.Parameters.ContainerResources != nil {+		ls := labels.Set(gen.Application.Parameters.ContainerResources.Labels).String()++		for i := range gen.ContainerResourcesSelectors {+			gen.ContainerResourcesSelectors[i].LabelSelector = ls+		}+	}++	list, err := gen.Generate()+	if err != nil {+		return err+	}++	for idx, listItem := range list.Items {+		listBytes, err := listItem.Marshal()+		if err != nil {+			return err+		}++		assetName := fmt.Sprintf("%s%d%s", "application-assets", idx, ".yaml")+		if err := o.Fs.WriteFile(assetName, listBytes); err != nil {+			return err+		}++		o.resources[assetName] = struct{}{}++		if te, ok := list.Items[idx].Object.(*redsky.Experiment); ok {+			o.experiment = &redsky.Experiment{}+			te.DeepCopyInto(o.experiment)+			break

dont break here

bradbeam

comment created time in 10 days

Pull request review commentthestormforge/optimize-controller

Minor clean up for repo rename

 **-!/api-!/controllers-!/internal-!/redskyapi-!/redskyctl-!/go.mod-!/go.sum-!/main.go

Not sure I'm following this change?

jgustie

comment created time in 10 days

Pull request review commentthestormforge/optimize-controller

Minor clean up for repo rename

 dockers:     dockerfile: redskyctl/Dockerfile brews:   - tap:-      owner: redskyops+      owner: thestormforge       name: homebrew-tap     folder: Formula     commit_author:       name: Butch Masters       email: butch@carbonrelay.com-    homepage: "https://redskyops.dev/"-    description: Kubernetes Exploration+    homepage: "https://www.stormforge.io/"+    description: Release with Confidence

™️

jgustie

comment created time in 10 days

push eventredskyops/redskyops-controller

Brad Beam

commit sha b32ab3bd514c6556bad9fd20d29bfab96b25457c

chore: Remove docs The docs have been migrated out of this repository, so we can remove the docs from here. Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha dc5cf3aa26e21ec828925ea62e1f5e77e5f77ac4

Merge pull request #231 from bradbeam/removedocs chore: Remove docs

view details

push time in 12 days

PR merged redskyops/redskyops-controller

chore: Remove docs

The docs have been migrated out of this repository, so we can remove the docs from here.

Signed-off-by: Brad Beam brad.beam@carbonrelay.com

+0 -2816

0 comment

53 changed files

bradbeam

pr closed time in 12 days

PR opened redskyops/redskyops-controller

chore: Remove docs

The docs have been migrated out of this repository, so we can remove the docs from here.

Signed-off-by: Brad Beam brad.beam@carbonrelay.com

+0 -2816

0 comment

53 changed files

pr created time in 12 days

push eventredskyops/redskyops-controller

Brad Beam

commit sha 864345cdb217b7ed9d7ac88c22b1680df914d31d

feat: Wait for prometheus deployment to be up Signed-off-by: Brad Beam <brad.beam@carbonrelay.com>

view details

Brad Beam

commit sha 0f62239789b10313a30e81b314efd6c5ba291427

Merge pull request #230 from bradbeam/waitfor feat: Wait for prometheus deployment to be up

view details

push time in 12 days

push eventredskyops/redskyops-controller

John

commit sha 45d541cb4190788babccb64289abc66b98c842a1

fixed prom query

view details

push time in 12 days

pull request commentredskyops/redskyops-controller

fix: Properly set GroupKindVersion on experiment resource

we'll just classify this under bugs that come up at 2am.

bradbeam

comment created time in 15 days

Pull request review commentredskyops/redskyops-controller

Include build metadata in UA string of pre-releases

 func userAgentString(product, comment string) string { 	if product == "" { 		return "" 	}+ 	// TODO Validate "product"+	ua := strings.Builder{}+	ua.WriteString(product)+	ua.WriteRune('/')+	ua.WriteString(strings.TrimPrefix(Version, "v"))++	// Build the comments using both the user supplied comments and some additional build information+	var comments []string++	// Only include build metadata for pre-release versions+	if strings.Contains(Version, "-") && BuildMetadata != "" {+		comments = append(comments, BuildMetadata)+	} +	// Clean white space and surrounding comment indicators+	comment = strings.TrimSpace(comment)+	comment = strings.TrimLeft(comment, "(")+	comment = strings.TrimRight(comment, ")") 	comment = strings.TrimSpace(comment)

should these be under the if statement below?

jgustie

comment created time in 16 days

PR opened redskyops/redskyops-controller

chore: Add test for NewJob

Simple/basic test to make sure the job is getting created properly.

Signed-off-by: Brad Beam brad.beam@carbonrelay.com

+101 -0

0 comment

1 changed file

pr created time in 16 days

more