How to divide Python files?

Question

I am building a CLI web scraping application in Python that should gather data for stock traders. It should do the following:

• call Reddit's API to get news about stocks.
• call Twitter's API to get tweets about stocks.
• call Yahoo's API to get price data about stocks.
• call Google Trends to get "popularity over time" for stocks.

Each one of those actions has its own imports. Also, all of those actions will be triggered by a CLI.

What makes sense to me is that each action will be structured in a single .py file of its own with probably a single class per file.

Then in the main.py would have a CLI class that initiates other classes according to the user's flags and arguments.

• Is my logic correct here?
• What other things should I be considering?
• What directories should I include those files in and what other directories should I be having?

Answer 1

Your overall approach is sensible; but hard to judge without seeing actual code.

For a small app, there's nothing wrong with putting everything in one source file. As the app grows, if it helps you to organize and read your code by having multiple source files, or multiple separately imported modules, that's a good thing to do. However there's nothing in Python to dictate when (or if) you make that transition.

Other things to consider:

• You can simplify your life by using the right CLI library. Fire ( https://github.com/google/python-fire ) is a good one and does a lot of things automatically. You might find having all your classes in one file, or just importing all your classes into a main file, gives you a good basic CLI.

• If your CLI will be more sophisticated, have a look at Click ( https://click.palletsprojects.com/en/7.x/ )

• The way Python deals with imports within a project structure is not obvious. It depends on how you invoke the module / script (eg imports that work when running a module eg via python -m MODULE or an import MODULE statement) will often fail when running the very same code as a script via python /PATH/TO/SCRIPT.py . It's hard to summarize this in a short post, but it requires you to be aware of how you package your code as scripts vs. modules vs. separately installable packages; and requires you to be aware of how your code will be executed. This is often a hard thing to learn when you're new to the Python environment. I recommend you read a tutorial such as ( https://click.palletsprojects.com/en/7.x/ ).