如何从python脚本运行snscrape命令？

Question

I'm trying to download some tweets with snscrape . After installing, I can run a command like the following to download a few tweets:

snscrape --jsonl --max-results 4 twitter-search "#SherlockHolmes since:2015-01-01 until:2015-01-15" > sherlock_tweets.json

Now I want to run this command from within a python script. As I understand it, the way to do this is using the subprocess.run method. I use the following code to run the command from python:

import subprocess

# Running this in a terminal works
cmd = '''snscrape --jsonl --max-results 4 twitter-search "#SherlockHolmes since:2015-01-01 until:2015-01-15" > sherlock_tweets.json'''
arglist = cmd.split(" ")

process = subprocess.run(arglist, shell=True)

Running this, however, gives the following error.

usage: snscrape [-h] [--version] [-v] [--dump-locals] [--retry N] [-n N] [-f FORMAT | --jsonl] [--with-entity] [--since DATETIME] [--progress]
                {telegram-channel,weibo-user,vkontakte-user,instagram-user,instagram-hashtag,instagram-location,twitter-thread,twitter-search,reddit-user,reddit-subreddit,reddit-search,facebook-group,twitter-user,twitter-hashtag,twitter-list-posts,facebook-user,facebook-community,twitter-profile}
                ...
snscrape: error: the following arguments are required: scraper

Why is the behaviour not the same in these two cases? How do I accomplish running the command from a python script, getting the exact same behaviour as I would entering it in a terminal?

Answer 1

I don't know if you found the solution, but I ran this code and that worked for me :

import pandas as pd
import snscrape.modules.twitter as sntwitter

tweet_collection = pd.DataFrame({
'Username':[],
'Date'=[],
'Likes'=[],
'Content'=[]})

for tweet in sntwitter.TwitterSearchScraper(f'since:{date_beg} until:{date_end} from:{twitter_account}').get_items():
    tweets_collection = tweets_candidats.append({
        "Username":tweet.user.username,
        "Date":tweet.date,
        "Tweet":tweet.content,
        "Likes":tweet.likeCount,},ignore_index=True)
tweets_candidats.to_csv('Path/file.csv')

You can find more detail in the code on git hub

Twitter snscrape arguments