散景不能用SQL数据解释正确的比例

Question

Explored lots of various solutions here but not finding one that works. I'm using sqlite and pandas to read data from a SQL database, but Bokeh doesn't like the date. I've tried conversions to datetime, unixepoch, etc. and they all seem to yield the same result.

EDIT: Here's the full code:

from os.path import dirname, join

import pandas as pd
import pandas.io.sql as psql
import numpy as np
import sqlite3
import os

from math import pi

from bokeh.plotting import figure, output_file, show 
from bokeh.io import output_notebook, curdoc
from bokeh.models import ColumnDataSource, Div, DatetimeTickFormatter
from bokeh.models.widgets import Slider, Select, RadioButtonGroup
from bokeh.layouts import layout, widgetbox

import warnings
import datetime

warnings.filterwarnings('ignore')


## Set up the SQL Connection

conn = sqlite3.connect('/Users/<>/Documents/python_scripts/reptool/reptool_db')
c = conn.cursor()

## Run the SQL

proj = pd.read_sql(
                       """

                        SELECT  

                        CASE WHEN df is null THEN ds ELSE df END AS 'projdate',
                        CASE WHEN yhat is null THEN y ELSE yhat END AS 'projvol',
                        strftime('%Y',ds) as 'year'

                        FROM forecast 

                        LEFT JOIN actuals 
                        ON forecast.ds = actuals.df 

                       """, con=conn)


# HTML index page and inline CSS stylesheet

desc = Div(text=open("/Users/<>/Documents/python_scripts/reptool/description.html").read(), width=800)


## Rename Columns and create list sets

proj.rename(columns={'projdate': 'x', 'projvol': 'y'}, inplace=True)

x=list(proj['x'])
y=list(proj['y'])

# proj['projdate'] = [datetime.datetime.strptime(x, "%Y-%m-%d").date() for x in proj['projdate']]

# Create input controls


radio_button_group = RadioButtonGroup(
        labels=["Actuals", "Forecast","FY Projection"], active=0)

min_year = Slider(title="Period Start", start=2012, end=2018, value=2013, step=1)
max_year = Slider(title="Period End", start=2012, end=2018, value=2017, step=1)

## Declare systemic source

source = ColumnDataSource(data=dict(x=[], y=[], year=[]))


## Bokeh tools

TOOLS="pan,wheel_zoom,box_zoom,reset,xbox_select"

## Set up plot


p = figure(title="REP Forecast", plot_width=900, plot_height=300, tools=TOOLS, x_axis_label='date', x_axis_type='datetime', y_axis_label='volume', active_drag="xbox_select")

p.line(x=proj.index, y=y, line_width=2, line_alpha=0.6)


p.xaxis.major_label_orientation = pi/4

# p.xaxis.formatter = DatetimeTickFormatter(seconds=["%Y:%M"],
#                                             minutes=["%Y:%M"],
#                                             minsec=["%Y:%M"],
#                                             hours=["%Y:%M"])

# axis map


# definitions

def select_rep():
    selected = proj[
        (proj.year >= min_year.value) &
        (proj.year >= max_year.value)
    ]
    return selected

def update():
    proj = select_rep()
    source.data = dict(
        year=proj["year"]   
    )



controls = [min_year, max_year]
for control in controls:
    control.on_change('value', lambda attr, old, new: update())


sizing_mode = 'fixed'  # 'scale_width' also looks nice with this example

## Build the html page and inline CSS
inputs = widgetbox(*controls)
l = layout([
    [desc],
    [p],
    [inputs],
], )

# update()

curdoc().add_root(l)
curdoc().title = "REP"

The SQLite output in Terminal.app looks like this:

SQL

The result is, that the x-axis displays in milliseconds. Also, the y-axis is showing up as exponential notation:

Bokeh Plot

The issue seems somehow related to pandas use of indexing, and thus I can't reference "x" here. I rename the columns and force list sets which, by themselves, will print correctly... and should therefore plot into the line properly but as you'll see below, they don't:

proj.rename(columns={'projdate': 'x', 'projvol': 'y'}, inplace=True)

x=list(proj['x'])
y=list(proj['y'])

To get the line to render in Bokeh, I have to pass it the index because passing it anything else doesn't seem to get the glyph to render. So currently I have this:

p = figure(title="REP Forecast", plot_width=900, plot_height=300, tools=TOOLS, x_axis_label='date', x_axis_type='datetime', y_axis_label='volume', active_drag="xbox_select")

p.line(x=proj.index, y=y, line_width=2, line_alpha=0.6)

Tried converting to unixepoch in the SQL, same result. Tried converting to unixepoch in the data, same result. Tried using DateTimeTickFormatter, just shows all 5-6 years as one year (thinking it's just displaying the milliseconds as years rather than changing them from milliseconds to days.

I've looked here and in github, up and down, and tried different things but ultimately I can't find one working example where the source is a sql query not a csv.

Answer 1

None of these things have anything to do with SQL, Bokeh only cares about the data that you give it, not where it came from. You have specified that you want a datetime axis on the x-axis:

x_axis_type='datetime'

So, Bokeh will set up the plot with a ticker that picks "nice" values on a datetime scale, and with a tick formatter that displays tick locations as formatted dates. What is important, however, is that the data coordinates are in the appropriate units, which are floating point milliseconds since epoch .

You can provide x values directly in these units, but Bokeh will also automatically convert common datetime types (eg python stdlib, numpy, or pandas) to the right units automatically. So the easiest thing for you to do is pass a column of datetime values as the x values to line .

To be clear, this statement:

To render the line in Bokeh, it has to use the index

is incorrect. You can pass any dataframe column you like as the x-values, and I am suggesting you pass a column of datetimes.

Answer 2

I changed a line of the SQL to:

CASE WHEN df is null THEN strftime('%Y',ds) ELSE strftime('%Y',df) END AS 'projdate',

However, when I try expanding that specifier to %Y-%m-%d %H-%m-%s it just reads it as a string all over again.

And also by re-importing the data I was able to pass the date through here without using Index:

p.line(x=x, y=y, line_width=2, line_alpha=0.6)

But then I get this weird output: link .

So it's clear that it can read the year, but I need to pass through the full date to display the time series forecast. And it's still displaying the dates and y-values in the incorrect scale, regardless.

Going to noodle on this some more but if anyone has other suggestions, I'm thankful.

Answer 3

SOLVED the datetime problem. Added this after the SQL query:

proj['projdate'] = proj['projdate'].astype('datetime64[ns]')

Which in turn yields this:

Bokeh Plot

Still got a problem with the x-axis but since that's a straight numerical value, x_axis_type should fix it.

So far the working code looks like this (again, still iterating to add other controls but everything about the Bokeh plot itself works as intended):

# main.py

# created by:           <>
# version:              0.1.2
# created date:         07-Aug-2018
# modified date:        09-Aug-2018

from os.path import dirname, join

import pandas as pd
import pandas.io.sql as psql
import numpy as np
import sqlite3
import os

from math import pi

from bokeh.plotting import figure, output_file, show 
from bokeh.io import output_notebook, curdoc
from bokeh.models import ColumnDataSource, Div, DatetimeTickFormatter
from bokeh.models.widgets import Slider, Select, RadioButtonGroup
from bokeh.layouts import layout, widgetbox

import warnings
import datetime

warnings.filterwarnings('ignore')


## Set up the SQL Connection

conn = sqlite3.connect('/Users/<>/Documents/python_scripts/reptool/reptool_db')
c = conn.cursor()

## Run the SQL

proj = pd.read_sql(
                       """

                        SELECT  

                        CASE WHEN df is null THEN strftime('%Y-%m-%d',ds) ELSE strftime('%Y-%m-%d',df) END AS 'projdate',
                        CASE WHEN yhat is null THEN y ELSE yhat END AS 'projvol',
                        strftime('%Y',ds) as 'year'

                        FROM forecast 

                        LEFT JOIN actuals 
                        ON forecast.ds = actuals.df 

                       """, con=conn)

proj['projdate'] = proj['projdate'].astype('datetime64[ns]')


# HTML index page and inline CSS stylesheet

desc = Div(text=open("/Users/<>/Documents/python_scripts/reptool/description.html").read(), width=800)


## Rename Columns and create list sets

proj.rename(columns={'projdate': 'x', 'projvol': 'y'}, inplace=True)

x=list(proj['x'])
y=list(proj['y'])


# Create input controls


radio_button_group = RadioButtonGroup(
        labels=["Actuals", "Forecast","FY Projection"], active=0)

min_year = Slider(title="Period Start", start=2012, end=2018, value=2013, step=1)
max_year = Slider(title="Period End", start=2012, end=2018, value=2017, step=1)

## Declare systemic source

source = ColumnDataSource(data=dict(x=[], y=[], year=[]))


## Bokeh tools

TOOLS="pan,wheel_zoom,box_zoom,reset,xbox_select"

## Set up plot


p = figure(title="REP Forecast", plot_width=900, plot_height=300, tools=TOOLS, x_axis_label='date', x_axis_type='datetime', y_axis_label='volume', active_drag="xbox_select")

p.line(x=x, y=y, line_width=2, line_alpha=0.6)


p.xaxis.major_label_orientation = pi/4

# p.xaxis.formatter = DatetimeTickFormatter(seconds=["%Y:%M"],
#                                             minutes=["%Y:%M"],
#                                             minsec=["%Y:%M"],
#                                             hours=["%Y:%M"])

# axis map


# definitions

def select_rep():
    selected = proj[
        (proj.year >= min_year.value) &
        (proj.year >= max_year.value)
    ]
    return selected

def update():
    proj = select_rep()
    source.data = dict(
        year=proj["year"]   
    )



controls = [min_year, max_year]
for control in controls:
    control.on_change('value', lambda attr, old, new: update())


sizing_mode = 'fixed'  # 'scale_width' also looks nice with this example

## Build the html page and inline CSS
inputs = widgetbox(*controls)
l = layout([
    [desc],
    [p],
    [inputs],
], )

# update()

curdoc().add_root(l)
curdoc().title = "REP"