Python (Jupyter) -> Vega -> Kibana？

Question

So we want to leverage the data-science and visualization strengths of Python on our ELK data and then use the Elastic API to send the resulting visualization to update a dashboard.

There are plenty of resources discussing this, only one of which actually provided a working example, however they leveraged now deprecated security bypasses, more of a hack. Specifically, this refers to setting the index name to .kibana , which now results in the error:

AuthorizationException: AuthorizationException(403, 'security_exception', 'action [indices:data/write/bulk[s]] is unauthorized for user [elastic] with roles [superuser] on indices [.kibana_8.1.2_001,.kibana], this action is granted by the index privileges [create_doc,create,delete,index,write,all]')

We think this must be possible through normal API usage without disabling any security settings. We did try adding a user with every possible permission added, and it could not perform this action during our testing.

Here's the example referred to as well as the overarching project which inspired us to try this .

Note that Vega is now a default feature of Kibana rather than a plugin, so this workflow should be even more viable now.

So our code goes like this:

import eland as ed
import datetime
import altair as alt
import eland as ed
import json
import numpy as np
import matplotlib.pyplot as plt
import vega_datasets
from elasticsearch import Elasticsearch

cloud_id = "secret"
http_auth = ("username", "password")
es = Elasticsearch(cloud_id=cloud_id, http_auth=http_auth)

data = vega_datasets.data
pd_df = data.cars()
chart = alt.Chart(pd_df).mark_point().encode(
    x='Miles_per_Gallon',
    y='Horsepower'
).interactive()

def saveVegaVis(client, index, visName, altairChart, resultSize=100, timeField=True):
    chart_json = json.loads(altairChart.to_json())
    visState = {
      "type": "vega",
      "aggs": [],
      "params": {
        "spec": json.dumps(chart_json, sort_keys=True, indent=4, separators=(',', ': ')),
      },
      "title": visName
    }
    visSavedObject={
        "visualization" : {
          "title" : visName,
          "visState" : json.dumps(visState, sort_keys=True, indent=4, separators=(',', ': ')),
          "uiStateJSON" : "{}",
          "description" : "",
          "version" : 1,
          "kibanaSavedObjectMeta" : {
            "searchSourceJSON" : json.dumps({
              "query": {
                "language": "kuery",
                "query": ""
              },
              "filter": []
            }),
          }
        },
        "type" : "visualization",
        "references" : [ ],
        "migrationVersion" : {
          "visualization" : "8.0.0"
        },
        "updated_at" : datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S.000Z")
    }


    return client.index(index=index,id='visualization:'+visName,body=visSavedObject)

saveVegaVis(es, 'test_visuals', 'def-vega-cars-1', chart, resultSize=1000)

After executing this code, we get a success message:

ObjectApiResponse({'_index': 'test_visuals', '_id': 'visualization:def-vega-cars-1', '_version': 8, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1})

But within ELK, the resulting object is not treated like a visualization, it's treated like a normal index entry.

We want it to show up like:

But rather, we can only see it as a normal index entry, like so:

It seems to us that all of the traits for a visualization are there. To validate this, we exported a Vega visualization to observe the data structure (please excuse any oddities, the export leaves a lot of escape characters we tried to clean up):

{
    "attributes": {
        "description": "",
        "kibanaSavedObjectMeta": {
            "searchSourceJSON": {
                "query ":{
                    "query":"",
                    "language":"kuery"
                },
                "filter":[]
            }
        },
        "title": "TEST_VISUAL_PLZ_WORK",
        "uiStateJSON": "{}",
        "version": 1,
        "visState": {
        "title":"TEST_VISUAL_PLZ_WORK",
        "type":"vega",
        "aggs":[],
        "params":{
            "spec":" {
            "$schema": "https://vega.github.io/schema/vega/v3.json", n "width": 300, "height": 100, "data": [{
                n "name": "vals",
                n "values": [n {
                        "category": 50,
                        "count": 30
                    }, {
                        "category": 100,
                        "count": 80
                    }, {
                        "category": 150,
                        "count": 10
                    }, {
                        "category": 200,
                        "count": 50
                    }
                ]
            }], "marks": [{
                "type": "rect",
                "from": {
                    "data": "vals"
                },
                "encode": {
                    "update": {
                        "x": {
                            "field": "category"
                        },
                        "width": {
                            "value": 30
                        },
                        "y": {
                            "field": "count"
                        },
                        "y2": {
                            "value": 0
                        }
                    }
                }
            }]

        }
        "}}"
    },
    "coreMigrationVersion": "8.1.2",
    "id": "6e130cc0-b694-11ec-8df1-41f60ea92d87",
    "migrationVersion": {
        "visualization": "8.0.0"
    },
    "references": [],
    "type": "visualization",
    "updated_at": "2022-04-07T17:04:32.085Z",
    "version": "WzYxOTQsMl0="
} {
    "excludedObjects": [],
    "excludedObjectsCount": 0,
    "exportedCount": 1,
    "missingRefCount": 0,
    "missingReferences": []
}

And it seems our data structure matches this.

So are we going about this the wrong way? Is there some minor mistake? Is it possible to do (without disabling security protocols / 'hacking' it)?

Answer 1

Looking at the example it looks like they drop it into a.kibana index which I know is a special index relating to kibana. You may want to change the index to that and see what happens.