ClassNotFoundException while using jdbcio connector of apache beam

Question

I am trying to read some data from mysql using jdbcio connector available in apache beam. As I want to use the default expansion service, as per the documentation: https://beam.apache.org/releases/pydoc/2.24.0/apache_beam.io.jdbc.html I installed the Java Runtime and made sure that java command is available.And then I tried to run the following code.

I am using Python version 3.9.6 and apache beam version 2.35.0

from apache_beam.io.jdbc import ReadFromJdbc
import apache_beam as beam

with beam.Pipeline() as p:
  result = (
      p
      | 'Read from jdbc' >> ReadFromJdbc(
          table_name='staff',
          driver_class_name='com.mysql.jdbc.Driver',
          jdbc_url='jdbc:mysql://localhost:3306/collegespace',
          username='root',
          password='root',
          
      )
      | 'Printing' >> beam.Map(print)
      )

And I am getting the error as below:

Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\apache_beam\transforms\external.py", line 529, in _service
    yield stub
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\apache_beam\transforms\external.py", line 475, in expand
    raise RuntimeError(response.error)
RuntimeError: org.apache.beam.sdk.io.jdbc.BeamSchemaInferenceException: Failed to infer Beam schema
        at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.inferBeamSchema(JdbcIO.java:657)
        at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.expand(JdbcIO.java:632)
        at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.expand(JdbcIO.java:551)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
        at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
        at org.apache.beam.sdk.io.jdbc.JdbcSchemaIOProvider$JdbcSchemaIO$1.expand(JdbcSchemaIOProvider.java:132)
        at org.apache.beam.sdk.io.jdbc.JdbcSchemaIOProvider$JdbcSchemaIO$1.expand(JdbcSchemaIOProvider.java:110)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:499)
        at org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider.apply(ExpansionService.java:396)
        at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:515)
        at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:591)
        at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:219)
        at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
        at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
        at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
        at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.sql.SQLException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver'
        at org.apache.commons.dbcp2.DriverFactory.createDriver(DriverFactory.java:54)
        at org.apache.commons.dbcp2.BasicDataSource.createConnectionFactory(BasicDataSource.java:462)
        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:528)
        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734)
        at org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows.inferBeamSchema(JdbcIO.java:651)
        ... 21 more
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at org.apache.commons.dbcp2.DriverFactory.createDriver(DriverFactory.java:49)
        ... 25 more

Answer 1

Try below Steps:

1. Download the file from here: https://dbschema.com/jdbc-drivers/OracleJdbcDriver.zip
2. Extract the zip file in you path.
3. Set environment variable export CLASSPATH='/path/to/ojdbc11.jar' (Present in the zip file downloaded)

OR

3. Pass CLASSPATH param inside your ReadFromJdbc method as classpath=['/path/to/ojdbc11.jar']