Forward all issues for Kubeflow org; setup dev environment

* #57 is tracking setting up new staging and prod environments

  * This PR sets up a new staging (or dev environment)
  * We create a kustomize manifest for deploying the front end into that
    namespace
  * The staging environment is configured to use the dev instance of the
    issue label bot backend microservice (i.e the pubsub workers)
  * I created some python scripts to make it easier to setup the secrets.
  * The motivation for doing this was to test the changes to the front end

* Front end now forwards all issues for the kubeflow org to the backend

  * This is needed because we want to use multiple models for all Kubeflow
    repos kubeflow/code-intelligence#70

  * The backend should also be configured with logging to measure the impact
    of the predictions.

kubeflow/code-intelligence#104 is an a test issue showing that the bot is
working.

* Fix how keys are handled

  * For GOOGLE_APPLICATION_CREDENTIALS; depend on that environment variable
    being set and pointing to the file containing the private key;
    don't get the private key from an environment variable and then write it
    to a file.

* For the GitHub App private key; use an environment variable to point to
  the file containing the PEM key.

* Create a script to create the secrets.

* Flask app is running in dev namespace

  * create_secrets.py creates secrets needed for dev instance

deployment/Dockerfile

@@ -68,6 +68,7 @@ RUN pip install \
     tensorflow==1.12.0 \
     seldon-core==0.2.6
 
+
 COPY requirements.txt .
 RUN pip install -r requirements.txt
 COPY flask_app flask_app/

deployment/README.md

@@ -5,6 +5,21 @@ with mlbot.net.
 
 This is currently running on a GKE cluster.
 
+See [machine-learning-apps/Issue-Label-Bot#57](https://github.com/machine-learning-apps/Issue-Label-Bot/issues/57) for a log of how
+the service was deployed.
+
+To build a new image
+
+```
+skaffold build
+```
+
+Then to update the image
+
+```
+cd overlays/dev|prod
+kustomize edit set image gcr.io/github-probots/label-bot-frontend=gcr.io/github-probots/label-bot-frontend:${TAG}@${SHA}
+```
 
 ## github-probots
 
@@ -40,12 +55,18 @@ Deploying it
 
 There is a staging cluster for testing running in
 
-* **GCP project**: issue-label-bot-dev
-* **cluster**: github-mlapp-test
-* **namespace**: mlapp
+* **GCP project**: github-probots
+* **cluster**:  kf-ci-ml
+* **namespace**: label-bot-dev
 
 Deploying it
 
+1. Create the secrets
+
+
+
+TODO(jlewi): instructions below are outdated
+
 1. Create the deployment
 
    ```

deployment/base/deployment.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ml-github-app
+  labels:
+    app: ml-github-app
+spec:
+  replicas: 9
+  selector:
+    matchLabels:
+      app: ml-github-app
+  template:
+    metadata:
+      labels:
+        app: ml-github-app
+    spec:
+      containers:
+      - name: frontend
+        image: gcr.io/github-probots/label-bot-frontend
+        command: ["python", "app.py"]
+        workingDir: "/flask_app"
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 3000
+          initialDelaySeconds: 10
+          periodSeconds: 3
+        env:
+        - name: DATABASE_URL
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: DATABASE_URL
+        - name: WEBHOOK_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: WEBHOOK_SECRET
+        # The values for the Kubeflow kf-label-bot-dev application
+        # See kubeflow/code-intelligence#84. This is suitable
+        # for development but shouldn't be used in production       
+        - name: APP_ID
+          value: "50112"
+        # Pato the GitHub app PEM key
+        - name: GITHUB_APP_PEM_KEY
+          value: /var/secrets/github/kf-label-bot-dev.private-key.pem
+        # The GCP project and pubsub topic to publish to.
+        # Default to the test/dev topic
+        - name: GCP_PROJECT_ID
+          value: issue-label-bot-dev
+        - name: GCP_PUBSUB_TOPIC_NAME
+          value: TEST_event_queue
+        - name: GOOGLE_APPLICATION_CREDENTIALS
+          value: /var/secrets/google/user-gcp-sa.json        
+        - name: FLASK_ENV
+          value: production
+        - name: PORT
+          value: '3000'
+        - name: APP_URL
+          value: https://mlbot.net/
+        - name: authors
+          value: 'c'
+        ports:
+        - containerPort: 443
+        - containerPort: 80
+        - containerPort: 3000
+        volumeMounts:
+        - name: user-gcp-sa
+          mountPath: /var/secrets/google
+        - name: github-app
+          mountPath: /var/secrets/github
+      volumes:
+      - name: user-gcp-sa
+        secret:
+          secretName: user-gcp-sa
+      - name: github-app
+        secret:
+          secretName: github-app

deployment/base/ingress.yaml

@@ -0,0 +1,20 @@
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: frontend  
+  annotations:
+    # The ip and certificate name should be overwritten for each
+    # overlay and set to the correct values
+    kubernetes.io/ingress.global-static-ip-name: fake-ip
+    networking.gke.io/managed-certificates: fake-certificate
+spec:
+  backend:
+    serviceName: ml-github-app
+    servicePort: 3000
+  rules:
+  - http:
+      paths:
+      - path: /
+        backend:
+          serviceName: ml-github-app
+          servicePort: 3000

deployment/base/kustomization.yaml

@@ -0,0 +1,13 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namePrefix: label-bot-
+commonLabels:
+  app: label-bot
+  service: label-bot
+images:
+- name: gcr.io/github-probots/label-bot-frontend
+  newName: gcr.io/github-probots/label-bot-frontend
+resources:
+- deployment.yaml
+- service.yaml
+- ingress.yaml

deployment/base/service.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: ml-github-app
+  labels:
+    app: ml-github-app
+spec:
+  ports:
+  - port: 3000
+    protocol: TCP
+  selector:
+    app: ml-github-app
+  type: NodePort

deployment/overlays/dev/certificate.yaml

@@ -0,0 +1,7 @@
+apiVersion: networking.gke.io/v1beta1
+kind: ManagedCertificate
+metadata:
+  name: certificate
+spec:
+  domains:
+    - label-bot-dev.mlbot.net

deployment/overlays/dev/deployment.yaml

@@ -0,0 +1,40 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ml-github-app
+spec:
+  replicas: 1
+  template:
+    spec:
+      containers:
+      - name: frontend
+        env:
+        - name: DATABASE_URL
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: DATABASE_URL
+        - name: WEBHOOK_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: WEBHOOK_SECRET        
+        # The values for the Kubeflow kf-label-bot-dev application
+        # See kubeflow/code-intelligence#84. This is suitable
+        # for development but shouldn't be used in production       
+        - name: APP_ID
+          value: "50112"
+        # Path the GitHub app PEM key
+        - name: GITHUB_APP_PEM_KEY
+          value: /var/secrets/github/kf-label-bot-dev.private-key.pem
+        # The GCP project and pubsub topic to publish to should
+        # correspond to the production backend
+        - name: GCP_PROJECT_ID
+          value: issue-label-bot-dev
+        - name: GCP_PUBSUB_TOPIC_NAME
+          value: TEST_event_queue
+        - name: GOOGLE_APPLICATION_CREDENTIALS
+          value: /var/secrets/google/user-gcp-sa.json        
+        - name: FLASK_ENV
+          value: production
+

deployment/overlays/dev/ingress.yaml

@@ -0,0 +1,7 @@
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: frontend  
+  annotations:
+    kubernetes.io/ingress.global-static-ip-name: label-bot-dev
+    networking.gke.io/managed-certificates: certificate

deployment/overlays/dev/kustomization.yaml

@@ -0,0 +1,12 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+bases:
+- ../../base
+commonLabels:
+  environment: dev
+namespace: label-bot-dev
+resources:
+- certificate.yaml
+patchesStrategicMerge:
+- deployment.yaml
+- ingress.yaml

deployment/overlays/prod/deployment.yaml

@@ -0,0 +1,38 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ml-github-app
+spec:
+  replicas: 9
+    spec:
+      containers:
+      - name: frontend
+        env:
+        - name: DATABASE_URL
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: DATABASE_URL
+        - name: WEBHOOK_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: ml-app-inference-secret
+              key: WEBHOOK_SECRET
+        # The values for the Kubeflow issue-label-bot application
+        # that is available in the GitHub marketplace      
+        - name: APP_ID
+          value: "27079"
+        # Pato the GitHub app PEM key
+        - name: GITHUB_APP_PEM_KEY
+          value: /var/secrets/github/kf-label-bot-dev.private-key.pem
+        # The GCP project and pubsub topic to publish to should
+        # correspond to the production backend
+        - name: GCP_PROJECT_ID
+          value: issue-label-bot-dev
+        - name: GCP_PUBSUB_TOPIC_NAME
+          value: event_queue
+        - name: GOOGLE_APPLICATION_CREDENTIALS
+          value: /var/secrets/google/user-gcp-sa.json        
+        - name: FLASK_ENV
+          value: production
+

developer_guide.md

@@ -0,0 +1,19 @@
+# Developer guide
+
+1. You can deploy the front end using skaffold
+
+   ```
+   skaffold dev --cleanup=False
+   ```
+
+   * Your Kubernetes context should be set to using the `github-probots-dev` namespace
+   * This will continually rebuild and upate your code
+   * Skaffold's file sync feature is used to update the code in the image without rebuilding and
+     redeploying
+   * This makes redeploying very easy.
+
+1. To send a GitHub webhook event you can either open up an issue or you can use `scripts/send_request.py`
+
+   * The latter is useful because it avoids needing to open up a new GitHub issue
+
+     * Right now the bot is only designed to respond to issues opened events.